2023-11-15T04:45:54.117 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T04:45:54.118 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425 --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/orig.config.yaml 2023-11-15T04:45:54.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T04:45:54.262 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458425' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmNWvIMYMBqczIfLppAztcZA0wBWzxz1SL+1QQMOdvHgZKGWzS4nzHF+OM/LI44DKYrFMadiP6k+Em4HBFUAG0= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcj3y44ysMXUmndcxZQ8cxB+MnBdbRyL1lJw1EbK1de/q/daTKUJBoA5iDZMVZA2nrRt7+eyUqkASeKsHiNICE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T04:45:54.263 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T04:45:54.263 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T04:45:54.264 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T04:45:54.266 INFO:teuthology.task.internal:Checking packages... 2023-11-15T04:45:54.294 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T04:45:54.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T04:45:54.295 INFO:teuthology.packaging:ref: None 2023-11-15T04:45:54.296 INFO:teuthology.packaging:tag: None 2023-11-15T04:45:54.296 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:45:54.296 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:45:54.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T04:45:54.513 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T04:45:54.513 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T04:45:54.521 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T04:45:54.522 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T04:45:54.535 INFO:teuthology.task.internal:Saving configuration 2023-11-15T04:45:54.555 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T04:45:54.565 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T04:45:54.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:40:59.062198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmNWvIMYMBqczIfLppAztcZA0wBWzxz1SL+1QQMOdvHgZKGWzS4nzHF+OM/LI44DKYrFMadiP6k+Em4HBFUAG0='} 2023-11-15T04:45:54.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:40:59.064060', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcj3y44ysMXUmndcxZQ8cxB+MnBdbRyL1lJw1EbK1de/q/daTKUJBoA5iDZMVZA2nrRt7+eyUqkASeKsHiNICE='} 2023-11-15T04:45:54.608 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T04:45:54.616 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T04:45:54.617 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T04:45:54.617 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T04:45:54.713 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45b4f18b80>, signals=[15]) 2023-11-15T04:45:54.714 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T04:45:54.722 INFO:teuthology.task.internal:Opening connections... 2023-11-15T04:45:54.723 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-11-15T04:45:54.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:45:54.793 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2023-11-15T04:45:54.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:45:54.859 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T04:45:54.871 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-11-15T04:45:54.894 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-11-15T04:45:54.895 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-11-15T04:45:54.955 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2023-11-15T04:45:54.956 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2023-11-15T04:45:54.956 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2023-11-15T04:45:54.956 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:45:54.956 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2023-11-15T04:45:54.956 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:45:54.957 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:45:54.957 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-11-15T04:45:54.957 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:45:54.957 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:45:54.957 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:45:54.958 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:45:54.958 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:45:54.959 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-11-15T04:45:54.990 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2023-11-15T04:45:55.012 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2023-11-15T04:45:55.013 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2023-11-15T04:45:55.073 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2023-11-15T04:45:55.074 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="8" 2023-11-15T04:45:55.074 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2023-11-15T04:45:55.074 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2023-11-15T04:45:55.075 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="8" 2023-11-15T04:45:55.075 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el8" 2023-11-15T04:45:55.075 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T04:45:55.075 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2023-11-15T04:45:55.075 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T04:45:55.076 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2023-11-15T04:45:55.076 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T04:45:55.076 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T04:45:55.076 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T04:45:55.077 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2023-11-15T04:45:55.098 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T04:45:55.109 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T04:45:55.241 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T04:45:55.242 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:45:55.245 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2023-11-15T04:45:55.265 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T04:45:55.275 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T04:45:55.276 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:45:55.306 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2023-11-15T04:45:55.340 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T04:45:55.400 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T04:45:55.419 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T04:45:55.425 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T04:45:55.426 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:45:55.427 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T04:45:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:45:55.447 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-11-15T04:45:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:45:55.448 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2023-11-15T04:45:55.507 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:45:55.508 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-489.el8.x86_64 2023-11-15T04:45:55.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2023-11-15T04:45:55.510 INFO:teuthology.orchestra.run.smithi154.stdout:4.18.0-489.el8.x86_64 2023-11-15T04:45:55.510 INFO:teuthology.task.kernel:Running kernel on smithi154: 4.18.0-489.el8.x86_64 2023-11-15T04:45:55.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2023-11-15T04:45:59.510 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 22 MB/s | 34 MB 00:01 2023-11-15T04:45:59.697 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - AppStream 21 MB/s | 34 MB 00:01 2023-11-15T04:46:06.978 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 42 MB/s | 53 MB 00:01 2023-11-15T04:46:07.097 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - BaseOS 46 MB/s | 53 MB 00:01 2023-11-15T04:46:15.127 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2023-11-15T04:46:15.146 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-11-15T04:46:15.689 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-15T04:46:15.803 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-11-15T04:46:15.990 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-11-15T04:46:16.079 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T04:46:16.699 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 29 MB/s | 16 MB 00:00 2023-11-15T04:46:17.697 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-15T04:46:20.508 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 455 kB/s | 24 kB 00:00 2023-11-15T04:46:21.572 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 357 kB/s | 24 kB 00:00 2023-11-15T04:46:24.191 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:46:24.191 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:46:24.466 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:46:24.467 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:46:24.467 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:46:24.468 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:46:24.468 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:46:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:46:24.468 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:46:24.468 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:46:24.469 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:46:24.469 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:46:24.469 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:46:24.469 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:46:24.469 INFO:teuthology.orchestra.run.smithi154.stdout:Install 3 Packages 2023-11-15T04:46:24.470 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:46:24.470 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 89 M 2023-11-15T04:46:24.471 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 96 M 2023-11-15T04:46:24.471 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:46:24.926 INFO:teuthology.orchestra.run.smithi154.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 24 MB/s | 10 MB 00:00 2023-11-15T04:46:25.180 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T04:46:25.180 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T04:46:25.461 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:46:25.462 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:46:25.462 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:46:25.462 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:46:25.462 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:46:25.463 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T04:46:25.463 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:46:25.463 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T04:46:25.463 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T04:46:25.463 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:46:25.464 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:46:25.464 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:46:25.464 INFO:teuthology.orchestra.run.smithi086.stdout:Install 3 Packages 2023-11-15T04:46:25.464 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:46:25.465 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 89 M 2023-11-15T04:46:25.465 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2023-11-15T04:46:25.465 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:46:25.848 INFO:teuthology.orchestra.run.smithi086.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 29 MB/s | 10 MB 00:00 2023-11-15T04:46:25.859 INFO:teuthology.orchestra.run.smithi154.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 27 MB/s | 36 MB 00:01 2023-11-15T04:46:26.268 INFO:teuthology.orchestra.run.smithi154.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 25 MB/s | 43 MB 00:01 2023-11-15T04:46:26.268 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:46:26.269 INFO:teuthology.orchestra.run.smithi154.stdout:Total 50 MB/s | 89 MB 00:01 2023-11-15T04:46:26.632 INFO:teuthology.orchestra.run.smithi086.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 32 MB/s | 36 MB 00:01 2023-11-15T04:46:26.942 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:46:27.090 INFO:teuthology.orchestra.run.smithi086.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 27 MB/s | 43 MB 00:01 2023-11-15T04:46:27.091 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:46:27.091 INFO:teuthology.orchestra.run.smithi086.stdout:Total 55 MB/s | 89 MB 00:01 2023-11-15T04:46:27.354 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:46:27.355 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:46:27.768 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:46:28.193 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:46:28.193 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:46:28.580 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:46:28.610 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:46:29.428 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:46:31.374 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:46:31.974 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:46:32.192 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:46:32.777 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:46:35.557 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:46:35.940 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:46:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:46:36.727 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:46:39.373 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:46:39.668 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:46:40.157 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:46:40.424 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:01.891 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:01.892 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:47:01.892 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:47:02.177 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:02.202 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:02.637 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:02.637 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T04:47:02.637 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T04:47:02.666 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:02.666 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:47:02.667 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:47:02.667 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:47:02.667 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:47:02.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:47:02.667 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:47:02.810 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:47:03.023 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.024 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.024 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.024 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:47:03.025 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:47:03.025 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T04:47:03.025 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:47:03.025 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:47:03.236 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2023-11-15T04:47:03.270 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2023-11-15T04:47:03.271 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2023-11-15T04:47:03.271 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.02 2023-11-15T04:47:03.271 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 129.el8 2023-11-15T04:47:03.271 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2023-11-15T04:47:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:47:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:Group : System Environment/Base 2023-11-15T04:47:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 9265586 2023-11-15T04:47:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2023-11-15T04:47:03.272 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Relocations : (not relocatable) 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : CentOS BuildSystem 2023-11-15T04:47:03.273 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2023-11-15T04:47:03.274 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:47:03.274 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2023-11-15T04:47:03.274 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2023-11-15T04:47:03.274 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:47:03.274 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:47:03.275 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:47:03.275 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:47:03.275 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2023-11-15T04:47:03.275 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:47:03.275 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:47:03.277 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:47:03.277 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:47:03.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T04:47:03.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:47:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:47:03.387 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.387 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:47:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:47:03.531 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:47:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.746 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.746 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.746 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T04:47:03.747 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T04:47:03.747 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T04:47:03.747 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T04:47:03.747 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T04:47:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T04:47:03.957 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2023-11-15T04:47:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2023-11-15T04:47:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2023-11-15T04:47:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2023-11-15T04:47:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 129.el8 2023-11-15T04:47:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2023-11-15T04:47:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T04:47:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2023-11-15T04:47:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9265586 2023-11-15T04:47:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2023-11-15T04:47:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T04:47:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T04:47:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T04:47:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T04:47:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2023-11-15T04:47:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : CentOS BuildSystem 2023-11-15T04:47:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2023-11-15T04:47:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T04:47:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2023-11-15T04:47:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2023-11-15T04:47:03.995 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:47:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T04:47:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T04:47:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T04:47:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2023-11-15T04:47:03.996 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:47:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T04:47:03.997 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T04:47:03.998 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-521.el8.x86_64 2023-11-15T04:47:03.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T04:47:04.906 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2023-11-15T04:47:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2023-11-15T04:47:07.012 INFO:teuthology.orchestra.run.smithi154.stderr:done 2023-11-15T04:47:07.014 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2023-11-15T04:47:07.050 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.lmON2QsCiE 2023-11-15T04:47:07.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lmON2QsCiE 2023-11-15T04:47:07.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.lmON2QsCiE 2023-11-15T04:47:07.251 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.lmON2QsCiE is 6KB 2023-11-15T04:47:07.299 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.lmON2QsCiE 2023-11-15T04:47:07.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:47:07.394 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:47:07.395 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:47:07.395 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:47:07.395 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:47:07.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:47:07.480 INFO:teuthology.orchestra.run.smithi086.stderr:done 2023-11-15T04:47:07.482 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-11-15T04:47:07.514 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.MWZwpHOmOx 2023-11-15T04:47:07.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MWZwpHOmOx 2023-11-15T04:47:07.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2023-11-15T04:47:07.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.MWZwpHOmOx 2023-11-15T04:47:07.733 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.MWZwpHOmOx is 6KB 2023-11-15T04:47:07.782 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.MWZwpHOmOx 2023-11-15T04:47:07.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T04:47:07.872 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T04:47:07.873 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T04:47:07.873 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T04:47:07.873 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T04:47:07.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T04:47:08.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2023-11-15T04:47:37.586 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T04:47:37.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:47:37.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:47:38.030 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T04:47:38.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-11-15T04:47:38.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:48:12.159 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-11-15T04:48:21.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:48:21.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:48:38.046 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T04:48:39.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-11-15T04:48:47.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-11-15T04:48:47.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:48:47.681 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-11-15T04:48:47.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-11-15T04:48:47.971 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:48:47.972 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-11-15T04:48:47.988 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:48:47.989 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:48:47.989 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:48:47.989 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:48:48.991 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T04:48:48.992 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:48:49.084 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2023-11-15T04:48:49.098 DEBUG:teuthology.parallel:result is None 2023-11-15T04:48:51.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:48:51.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:48:51.867 DEBUG:teuthology.orchestra.run.smithi154:> true 2023-11-15T04:48:52.166 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:48:52.166 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T04:48:52.167 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2023-11-15T04:48:52.183 INFO:teuthology.orchestra.run.smithi154.stdout:4.18.0-521.el8.x86_64 2023-11-15T04:48:52.184 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T04:48:52.184 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T04:48:52.184 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T04:48:53.185 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T04:48:53.186 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T04:48:53.283 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2023-11-15T04:48:53.296 DEBUG:teuthology.parallel:result is None 2023-11-15T04:48:53.297 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T04:48:53.304 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T04:48:53.305 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:48:53.308 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T04:48:53.326 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T04:48:53.333 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T04:48:53.341 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T04:48:53.341 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:48:53.368 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T04:48:53.394 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T04:48:53.401 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T04:48:53.402 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:48:53.432 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T04:48:53.460 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:48:53.470 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:48:53.474 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:48:53.485 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T04:48:53.486 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T04:48:53.493 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T04:48:53.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:48:53.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T04:48:53.548 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T04:48:53.559 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T04:48:53.559 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:48:53.588 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T04:48:53.605 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:48:53.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:48:53.747 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:48:53.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:48:53.873 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T04:48:53.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:48:53.939 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:48:53.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T04:48:54.029 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:48:54.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T04:48:54.156 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T04:48:54.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T04:48:54.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-11-15T04:48:54.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2023-11-15T04:48:54.270 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:48:54.330 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T04:48:54.750 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T04:48:54.759 INFO:teuthology.task.internal:Starting timer... 2023-11-15T04:48:54.759 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T04:48:54.770 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T04:48:54.780 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T04:48:54.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-11-15T04:48:54.913 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2023-11-15T04:48:54.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2023-11-15T04:48:55.082 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2023-11-15T04:48:55.084 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T04:48:55.084 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-11-15T04:48:55.111 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-11-15T04:48:55.111 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2023-11-15T04:48:55.407 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2023-11-15T04:48:55.407 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2023-11-15T04:48:55.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:48:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:48:55.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T04:48:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T04:48:55.468 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T04:48:55.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:48:55.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2023-11-15T04:48:55.596 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T04:48:55.606 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T04:48:55.607 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T04:48:55.625 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T04:48:55.627 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T04:56:27.321 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2023-11-15T04:56:27.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-11-15T04:56:27.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:56:27.394 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-11-15T04:56:27.464 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-11-15T04:56:27.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:56:27.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T04:56:27.532 DEBUG:teuthology.orchestra.run.smithi154:> true 2023-11-15T04:56:27.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2023-11-15T04:56:27.600 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T04:56:27.610 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T04:56:27.611 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:56:27.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:56:27.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T04:56:27.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T04:56:27.643 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:56:27.657 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:56:27.671 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T04:56:27.685 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T04:56:27.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-11-15T04:56:27.744 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-11-15T04:56:27.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2023-11-15T04:56:27.758 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:56:27.765 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2023-11-15T04:56:27.771 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:56:27.780 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T04:56:27.793 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T04:56:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-11-15T04:56:27.840 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:56:27.841 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-11-15T04:56:27.841 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.841 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.841 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.841 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.856 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2023-11-15T04:56:27.861 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T04:56:27.861 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2023-11-15T04:56:27.861 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.861 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.862 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.862 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T04:56:27.863 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:56:27.872 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:56:27.873 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2023-11-15T04:56:27.873 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:56:27.874 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2023-11-15T04:56:27.874 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:56:28.761 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:38 ago on Wed 15 Nov 2023 04:54:50 AM UTC. 2023-11-15T04:56:28.786 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Wed 15 Nov 2023 04:54:50 AM UTC. 2023-11-15T04:56:30.863 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:56:30.864 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:30.864 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:56:30.864 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:30.865 INFO:teuthology.orchestra.run.smithi154.stdout:Resetting modules: 2023-11-15T04:56:30.865 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools 2023-11-15T04:56:30.865 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:30.865 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:56:30.866 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:30.866 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:30.884 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:56:30.885 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:30.885 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:56:30.885 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:30.886 INFO:teuthology.orchestra.run.smithi086.stdout:Resetting modules: 2023-11-15T04:56:30.886 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools 2023-11-15T04:56:30.886 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:30.886 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:56:30.887 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:30.887 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:31.104 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:56:31.130 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:56:32.055 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Wed 15 Nov 2023 04:54:50 AM UTC. 2023-11-15T04:56:32.056 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Wed 15 Nov 2023 04:54:50 AM UTC. 2023-11-15T04:56:34.174 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:56:34.186 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:56:34.349 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:34.350 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:56:34.350 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:34.350 INFO:teuthology.orchestra.run.smithi086.stdout:Installing group/module packages: 2023-11-15T04:56:34.350 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:56:34.351 INFO:teuthology.orchestra.run.smithi086.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:56:34.351 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:56:34.351 INFO:teuthology.orchestra.run.smithi086.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:56:34.351 INFO:teuthology.orchestra.run.smithi086.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:56:34.351 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:56:34.352 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:56:34.352 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:56:34.352 INFO:teuthology.orchestra.run.smithi086.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:56:34.352 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:56:34.352 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:56:34.353 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:56:34.353 INFO:teuthology.orchestra.run.smithi086.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:56:34.353 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2023-11-15T04:56:34.353 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:56:34.353 INFO:teuthology.orchestra.run.smithi086.stdout:Installing module profiles: 2023-11-15T04:56:34.354 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools/common 2023-11-15T04:56:34.354 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:34.354 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling module streams: 2023-11-15T04:56:34.354 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools rhel8 2023-11-15T04:56:34.354 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:34.355 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:56:34.355 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:56:34.355 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2023-11-15T04:56:34.355 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:34.356 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 27 M 2023-11-15T04:56:34.356 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 89 M 2023-11-15T04:56:34.356 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:56:34.364 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:56:34.365 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:34.365 INFO:teuthology.orchestra.run.smithi154.stdout:Installing group/module packages: 2023-11-15T04:56:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T04:56:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T04:56:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T04:56:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T04:56:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T04:56:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T04:56:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T04:56:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T04:56:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T04:56:34.367 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:56:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T04:56:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T04:56:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T04:56:34.368 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2023-11-15T04:56:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T04:56:34.368 INFO:teuthology.orchestra.run.smithi154.stdout:Installing module profiles: 2023-11-15T04:56:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools/common 2023-11-15T04:56:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:34.369 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling module streams: 2023-11-15T04:56:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools rhel8 2023-11-15T04:56:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:34.369 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:56:34.370 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:56:34.370 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2023-11-15T04:56:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:34.371 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 27 M 2023-11-15T04:56:34.371 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 89 M 2023-11-15T04:56:34.371 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:56:34.812 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.8 MB/s | 750 kB 00:00 2023-11-15T04:56:34.863 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.1 MB/s | 1.0 MB 00:00 2023-11-15T04:56:34.888 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 501 kB/s | 38 kB 00:00 2023-11-15T04:56:34.947 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T04:56:34.990 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.3 MB/s | 1.0 MB 00:00 2023-11-15T04:56:35.015 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.6 MB/s | 750 kB 00:00 2023-11-15T04:56:35.030 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2023-11-15T04:56:35.066 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 505 kB/s | 38 kB 00:00 2023-11-15T04:56:35.099 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T04:56:35.106 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2023-11-15T04:56:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.3 MB/s | 178 kB 00:00 2023-11-15T04:56:35.206 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-15T04:56:35.242 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 370 kB/s | 25 kB 00:00 2023-11-15T04:56:35.265 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 10 MB/s | 3.9 MB 00:00 2023-11-15T04:56:35.290 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-pyxdg-0.25-16.el8.noarch.rpm 511 kB/s | 94 kB 00:00 2023-11-15T04:56:35.384 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-15T04:56:35.409 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 562 kB/s | 94 kB 00:00 2023-11-15T04:56:35.424 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 16 MB/s | 2.5 MB 00:00 2023-11-15T04:56:35.449 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 331 kB/s | 53 kB 00:00 2023-11-15T04:56:35.467 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.8 MB/s | 3.9 MB 00:00 2023-11-15T04:56:35.499 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2023-11-15T04:56:35.543 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 702 kB/s | 53 kB 00:00 2023-11-15T04:56:35.616 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 23 MB/s | 8.1 MB 00:00 2023-11-15T04:56:35.616 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:56:35.617 INFO:teuthology.orchestra.run.smithi086.stdout:Total 21 MB/s | 27 MB 00:01 2023-11-15T04:56:35.626 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 12 MB/s | 2.5 MB 00:00 2023-11-15T04:56:35.651 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 378 kB/s | 41 kB 00:00 2023-11-15T04:56:35.743 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 23 MB/s | 8.1 MB 00:00 2023-11-15T04:56:35.744 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:56:35.744 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 27 MB 00:01 2023-11-15T04:56:35.883 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:56:35.904 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:56:35.905 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:56:36.013 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:56:36.035 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:56:36.035 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:56:36.169 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:56:36.172 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:56:36.296 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:56:36.299 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:56:36.852 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:56:36.992 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:56:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:56:37.208 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T04:56:37.309 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:56:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T04:56:37.533 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:56:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T04:56:37.786 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:56:37.846 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:56:38.384 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:56:38.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T04:56:39.362 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:56:39.435 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:56:39.604 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:56:39.661 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T04:56:39.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:56:39.908 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T04:56:40.080 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:56:40.148 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T04:56:40.488 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:56:40.549 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T04:56:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:56:41.462 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:56:41.630 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:56:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T04:56:41.825 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:56:41.893 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:56:42.241 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:56:42.241 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:56:42.242 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:56:42.242 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:56:42.242 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:56:42.243 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:56:42.243 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:56:42.243 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:56:42.243 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:56:42.244 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:56:42.244 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:56:42.244 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:56:42.245 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:56:42.302 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T04:56:42.302 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T04:56:42.302 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T04:56:42.302 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T04:56:42.303 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T04:56:42.303 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T04:56:42.303 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T04:56:42.303 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T04:56:42.304 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T04:56:42.304 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T04:56:42.304 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T04:56:42.304 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T04:56:42.304 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T04:56:43.008 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:56:43.009 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:43.009 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:56:43.009 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.009 INFO:teuthology.orchestra.run.smithi086.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.010 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.010 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:56:43.010 INFO:teuthology.orchestra.run.smithi086.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.011 INFO:teuthology.orchestra.run.smithi086.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.011 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.011 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:56:43.011 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:56:43.012 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.012 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.012 INFO:teuthology.orchestra.run.smithi086.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.012 INFO:teuthology.orchestra.run.smithi086.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:56:43.012 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:56:43.013 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:56:43.100 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T04:56:43.101 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:43.101 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:56:43.101 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.102 INFO:teuthology.orchestra.run.smithi154.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.102 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.102 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T04:56:43.102 INFO:teuthology.orchestra.run.smithi154.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.103 INFO:teuthology.orchestra.run.smithi154.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.103 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.103 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T04:56:43.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T04:56:43.104 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.104 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T04:56:43.105 INFO:teuthology.orchestra.run.smithi154.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T04:56:43.105 INFO:teuthology.orchestra.run.smithi154.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T04:56:43.106 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:56:43.106 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:56:43.282 DEBUG:teuthology.parallel:result is None 2023-11-15T04:56:43.351 DEBUG:teuthology.parallel:result is None 2023-11-15T04:56:43.352 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T04:56:43.360 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T04:56:43.361 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2023-11-15T04:56:43.361 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:56:43.361 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2023-11-15T04:56:43.362 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T04:56:43.554 DEBUG:teuthology.parallel:result is None 2023-11-15T04:56:43.859 DEBUG:teuthology.parallel:result is None 2023-11-15T04:56:43.859 INFO:teuthology.run_tasks:Running task install... 2023-11-15T04:56:43.869 DEBUG:teuthology.task.install:project ceph 2023-11-15T04:56:43.870 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T04:56:43.870 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T04:56:43.871 INFO:teuthology.task.install:Using flavor: default 2023-11-15T04:56:43.877 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T04:56:43.878 INFO:teuthology.task.install:extra packages: [] 2023-11-15T04:56:43.878 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:56:43.879 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:56:43.879 INFO:teuthology.packaging:ref: None 2023-11-15T04:56:43.879 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:56:43.880 INFO:teuthology.packaging:branch: None 2023-11-15T04:56:43.880 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:56:44.076 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:56:44.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:56:44.080 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T04:56:44.080 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:56:44.081 INFO:teuthology.packaging:ref: None 2023-11-15T04:56:44.081 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:56:44.081 INFO:teuthology.packaging:branch: None 2023-11-15T04:56:44.082 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:56:44.277 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:56:44.278 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T04:56:44.496 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:56:44.496 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:56:44.501 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T04:56:44.501 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T04:56:44.546 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:56:44.546 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T04:56:44.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:56:44.550 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T04:56:44.550 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T04:56:44.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T04:56:44.579 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:56:44.579 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:56:44.579 INFO:teuthology.packaging:ref: None 2023-11-15T04:56:44.580 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:56:44.580 INFO:teuthology.packaging:branch: None 2023-11-15T04:56:44.580 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:56:44.580 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:56:44.586 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T04:56:44.586 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T04:56:44.586 INFO:teuthology.packaging:ref: None 2023-11-15T04:56:44.587 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T04:56:44.587 INFO:teuthology.packaging:branch: None 2023-11-15T04:56:44.587 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T04:56:44.587 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T04:56:44.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:56:44.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T04:56:44.733 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:56:44.740 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T04:56:44.801 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-11-15T04:56:44.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-11-15T04:56:44.808 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2023-11-15T04:56:44.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2023-11-15T04:56:45.085 INFO:teuthology.orchestra.run.smithi154.stdout:76 files removed 2023-11-15T04:56:45.089 INFO:teuthology.orchestra.run.smithi086.stdout:76 files removed 2023-11-15T04:56:45.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2023-11-15T04:56:45.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2023-11-15T04:56:45.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 862 kB/s | 74 kB 00:00 2023-11-15T04:56:45.628 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 712 kB/s | 74 kB 00:00 2023-11-15T04:56:45.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-11-15T04:56:45.744 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-11-15T04:56:45.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-11-15T04:56:45.853 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2023-11-15T04:56:47.567 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 20 MB/s | 34 MB 00:01 2023-11-15T04:56:47.629 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - AppStream 19 MB/s | 34 MB 00:01 2023-11-15T04:56:55.012 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - BaseOS 44 MB/s | 53 MB 00:01 2023-11-15T04:56:55.835 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 53 MB 00:02 2023-11-15T04:57:02.971 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-15T04:57:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 18 kB/s | 18 kB 00:00 2023-11-15T04:57:04.680 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras common packages 4.5 kB/s | 6.9 kB 00:01 2023-11-15T04:57:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-11-15T04:57:05.506 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - PowerTools 9.9 MB/s | 6.6 MB 00:00 2023-11-15T04:57:06.004 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.6 MB 00:00 2023-11-15T04:57:06.908 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 33 kB/s | 3.5 kB 00:00 2023-11-15T04:57:07.446 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T04:57:11.261 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2023-11-15T04:57:11.303 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 16 MB 00:04 2023-11-15T04:57:15.073 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 625 kB/s | 24 kB 00:00 2023-11-15T04:57:15.125 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 493 kB/s | 24 kB 00:00 2023-11-15T04:57:19.556 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:57:19.559 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:57:19.560 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:19.560 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:57:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:57:19.560 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-11-15T04:57:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:57:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:57:19.561 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:57:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:57:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:57:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:57:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:57:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:57:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:57:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:57:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:57:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:57:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:57:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:57:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:57:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:57:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:57:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:57:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:57:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:57:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:57:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:57:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:57:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:57:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:57:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:57:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:57:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:19.566 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:57:19.566 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:19.566 INFO:teuthology.orchestra.run.smithi086.stdout:Install 25 Packages 2023-11-15T04:57:19.567 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2023-11-15T04:57:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:19.567 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 59 M 2023-11-15T04:57:19.568 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:57:19.673 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:57:19.676 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:19.677 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:57:19.677 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:19.678 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:57:19.678 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T04:57:19.679 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2023-11-15T04:57:19.679 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:57:19.679 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T04:57:19.680 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:57:19.680 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T04:57:19.680 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T04:57:19.681 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T04:57:19.681 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T04:57:19.682 INFO:teuthology.orchestra.run.smithi154.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T04:57:19.682 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T04:57:19.682 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T04:57:19.683 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T04:57:19.683 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T04:57:19.683 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T04:57:19.684 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T04:57:19.684 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T04:57:19.684 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T04:57:19.685 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T04:57:19.685 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T04:57:19.685 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T04:57:19.686 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T04:57:19.686 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T04:57:19.687 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T04:57:19.687 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T04:57:19.688 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T04:57:19.688 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T04:57:19.688 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T04:57:19.689 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T04:57:19.689 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:19.690 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:57:19.690 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:19.690 INFO:teuthology.orchestra.run.smithi154.stdout:Install 25 Packages 2023-11-15T04:57:19.691 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2023-11-15T04:57:19.691 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:19.692 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 59 M 2023-11-15T04:57:19.693 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:57:19.951 INFO:teuthology.orchestra.run.smithi154.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 45 MB/s | 6.0 MB 00:00 2023-11-15T04:57:19.976 INFO:teuthology.orchestra.run.smithi154.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2023-11-15T04:57:20.019 INFO:teuthology.orchestra.run.smithi154.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T04:57:20.052 INFO:teuthology.orchestra.run.smithi154.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T04:57:20.144 INFO:teuthology.orchestra.run.smithi154.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2023-11-15T04:57:20.186 INFO:teuthology.orchestra.run.smithi154.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.0 MB/s | 44 kB 00:00 2023-11-15T04:57:20.220 INFO:teuthology.orchestra.run.smithi154.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.3 MB/s | 80 kB 00:00 2023-11-15T04:57:20.262 INFO:teuthology.orchestra.run.smithi154.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 5.0 MB/s | 213 kB 00:00 2023-11-15T04:57:20.304 INFO:teuthology.orchestra.run.smithi154.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 387 kB 00:00 2023-11-15T04:57:20.371 INFO:teuthology.orchestra.run.smithi154.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 5.4 MB/s | 368 kB 00:00 2023-11-15T04:57:20.396 INFO:teuthology.orchestra.run.smithi154.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T04:57:20.441 INFO:teuthology.orchestra.run.smithi086.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2023-11-15T04:57:20.466 INFO:teuthology.orchestra.run.smithi086.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-15T04:57:20.508 INFO:teuthology.orchestra.run.smithi086.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T04:57:20.530 INFO:teuthology.orchestra.run.smithi154.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T04:57:20.541 INFO:teuthology.orchestra.run.smithi086.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T04:57:20.634 INFO:teuthology.orchestra.run.smithi086.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2023-11-15T04:57:20.659 INFO:teuthology.orchestra.run.smithi086.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-15T04:57:20.676 INFO:teuthology.orchestra.run.smithi086.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2023-11-15T04:57:20.709 INFO:teuthology.orchestra.run.smithi086.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2023-11-15T04:57:20.743 INFO:teuthology.orchestra.run.smithi086.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T04:57:20.763 INFO:teuthology.orchestra.run.smithi154.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-11-15T04:57:20.776 INFO:teuthology.orchestra.run.smithi086.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T04:57:20.788 INFO:teuthology.orchestra.run.smithi154.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 370 kB/s | 145 kB 00:00 2023-11-15T04:57:20.793 INFO:teuthology.orchestra.run.smithi086.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.7 MB/s | 114 kB 00:00 2023-11-15T04:57:20.864 INFO:teuthology.orchestra.run.smithi154.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 584 kB/s | 44 kB 00:00 2023-11-15T04:57:20.947 INFO:teuthology.orchestra.run.smithi154.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 507 kB/s | 42 kB 00:00 2023-11-15T04:57:20.977 INFO:teuthology.orchestra.run.smithi086.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T04:57:20.997 INFO:teuthology.orchestra.run.smithi154.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 938 kB/s | 47 kB 00:00 2023-11-15T04:57:21.032 INFO:teuthology.orchestra.run.smithi154.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 2.3 MB/s | 80 kB 00:00 2023-11-15T04:57:21.082 INFO:teuthology.orchestra.run.smithi154.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 641 kB/s | 354 kB 00:00 2023-11-15T04:57:21.108 INFO:teuthology.orchestra.run.smithi154.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 756 kB/s | 259 kB 00:00 2023-11-15T04:57:21.133 INFO:teuthology.orchestra.run.smithi154.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 1.1 MB/s | 116 kB 00:00 2023-11-15T04:57:21.158 INFO:teuthology.orchestra.run.smithi154.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 2.3 MB/s | 59 kB 00:00 2023-11-15T04:57:21.191 INFO:teuthology.orchestra.run.smithi154.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 2.7 MB/s | 306 kB 00:00 2023-11-15T04:57:21.216 INFO:teuthology.orchestra.run.smithi154.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.6 MB/s | 181 kB 00:00 2023-11-15T04:57:21.242 INFO:teuthology.orchestra.run.smithi154.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 894 kB/s | 75 kB 00:00 2023-11-15T04:57:21.306 INFO:teuthology.orchestra.run.smithi086.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-11-15T04:57:21.326 INFO:teuthology.orchestra.run.smithi154.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 37 MB/s | 4.0 MB 00:00 2023-11-15T04:57:21.328 INFO:teuthology.orchestra.run.smithi086.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 272 kB/s | 145 kB 00:00 2023-11-15T04:57:21.384 INFO:teuthology.orchestra.run.smithi154.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-11-15T04:57:21.386 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:57:21.386 INFO:teuthology.orchestra.run.smithi154.stdout:Total 34 MB/s | 59 MB 00:01 2023-11-15T04:57:21.412 INFO:teuthology.orchestra.run.smithi086.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 528 kB/s | 44 kB 00:00 2023-11-15T04:57:21.455 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:57:21.503 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:57:21.504 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:57:21.504 INFO:teuthology.orchestra.run.smithi086.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 461 kB/s | 42 kB 00:00 2023-11-15T04:57:21.545 INFO:teuthology.orchestra.run.smithi086.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-11-15T04:57:21.587 INFO:teuthology.orchestra.run.smithi086.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.9 MB/s | 80 kB 00:00 2023-11-15T04:57:21.621 INFO:teuthology.orchestra.run.smithi086.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 550 kB/s | 354 kB 00:00 2023-11-15T04:57:21.663 INFO:teuthology.orchestra.run.smithi086.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 722 kB/s | 259 kB 00:00 2023-11-15T04:57:21.838 INFO:teuthology.orchestra.run.smithi086.stdout:(21/27): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2023-11-15T04:57:21.853 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:57:21.856 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:57:21.863 INFO:teuthology.orchestra.run.smithi086.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-11-15T04:57:21.889 INFO:teuthology.orchestra.run.smithi086.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-11-15T04:57:21.951 INFO:teuthology.orchestra.run.smithi086.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 865 kB/s | 75 kB 00:00 2023-11-15T04:57:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2023-11-15T04:57:22.082 INFO:teuthology.orchestra.run.smithi086.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.0 MB 00:00 2023-11-15T04:57:22.348 INFO:teuthology.orchestra.run.smithi086.stdout:(27/27): fmt-6.2.1-1.el8.x86_64.rpm 153 kB/s | 116 kB 00:00 2023-11-15T04:57:22.352 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:57:22.353 INFO:teuthology.orchestra.run.smithi086.stdout:Total 21 MB/s | 59 MB 00:02 2023-11-15T04:57:22.418 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:57:22.466 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:57:22.467 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:57:22.825 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:57:22.827 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:57:23.010 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:57:23.249 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:57:23.471 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:57:23.670 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:57:23.939 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:57:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:57:24.170 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T04:57:24.320 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T04:57:24.417 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:24.570 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:57:24.675 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:57:24.879 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:24.929 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:24.980 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T04:57:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:57:25.220 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:57:25.317 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:57:25.712 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:25.730 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:57:25.777 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:25.903 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T04:57:25.985 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:26.153 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:57:26.192 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:57:26.378 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:57:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:57:26.561 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T04:57:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:57:26.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:26.767 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:26.804 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:57:26.941 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T04:57:26.986 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:57:27.063 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:57:27.128 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:57:27.191 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T04:57:27.359 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:57:27.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:57:27.520 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:57:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:57:27.767 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T04:57:27.870 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:57:27.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:57:28.086 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:57:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:57:28.371 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:57:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T04:57:28.578 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:57:28.580 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:57:28.680 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T04:57:28.769 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:57:28.839 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:57:28.865 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T04:57:29.038 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:57:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T04:57:29.219 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:57:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:57:29.340 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T04:57:29.411 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:57:29.544 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T04:57:29.636 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:57:29.710 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T04:57:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T04:57:30.137 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T04:57:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:32.648 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:33.286 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:33.549 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:57:33.586 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T04:57:33.659 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:57:33.810 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:57:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T04:57:34.110 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:52.418 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-11-15T04:57:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-11-15T04:57:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-11-15T04:57:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-11-15T04:57:52.420 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2023-11-15T04:57:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2023-11-15T04:57:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2023-11-15T04:57:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /run 2023-11-15T04:57:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2023-11-15T04:57:52.451 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2023-11-15T04:57:52.451 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2023-11-15T04:57:52.451 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2023-11-15T04:57:52.451 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2023-11-15T04:57:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:53.763 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:57:53.778 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:57:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:57:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:57:53.842 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T04:57:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:57:54.027 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:57:54.027 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:54.059 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T04:57:54.059 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:54.727 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:54.728 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:57:54.728 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:57:54.728 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:57:54.729 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:54.729 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:57:54.729 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:54.729 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:57:54.730 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:57:54.730 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:54.730 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:57:54.730 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:57:54.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:57:54.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:57:54.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:57:54.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:57:54.731 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:57:54.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:57:54.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:57:54.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:57:54.732 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:57:54.733 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:57:54.733 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:57:54.733 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:57:54.733 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:57:54.733 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:57:54.734 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:54.734 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:57:54.734 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T04:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T04:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T04:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T04:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T04:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T04:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T04:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T04:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T04:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T04:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T04:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T04:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T04:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T04:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T04:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T04:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T04:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T04:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T04:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T04:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T04:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T04:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T04:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T04:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T04:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T04:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T04:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T04:57:55.937 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:55.937 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:55.938 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2023-11-15T04:57:55.938 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.938 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:57:55.938 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.939 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.939 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.939 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.939 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:57:55.940 INFO:teuthology.orchestra.run.smithi154.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:57:55.940 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:57:55.940 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:57:55.940 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.941 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:57:55.941 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:57:55.941 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:57:55.941 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.941 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:57:55.942 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.942 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:57:55.942 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:57:55.942 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:57:55.943 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.943 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.943 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.943 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:57:55.943 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.944 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.944 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:55.944 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:55.944 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:57:56.022 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T04:57:56.022 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:56.022 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-11-15T04:57:56.023 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.023 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:57:56.023 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.023 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.024 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.024 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.024 INFO:teuthology.orchestra.run.smithi086.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T04:57:56.024 INFO:teuthology.orchestra.run.smithi086.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T04:57:56.025 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T04:57:56.025 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T04:57:56.025 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.025 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T04:57:56.026 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T04:57:56.026 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T04:57:56.026 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.026 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T04:57:56.026 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.027 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T04:57:56.027 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T04:57:56.027 INFO:teuthology.orchestra.run.smithi086.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T04:57:56.027 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.027 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T04:57:56.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.028 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.029 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T04:57:56.029 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:56.029 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:57:56.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2023-11-15T04:57:56.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2023-11-15T04:57:57.045 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:42 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:57:57.124 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:42 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:57:58.708 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:57:58.709 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:58.710 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:57:58.710 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:58.710 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:57:58.710 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:57:58.710 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:57:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:57:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:57:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:57:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:57:58.711 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:57:58.712 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:58.712 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:57:58.712 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:57:58.712 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-11-15T04:57:58.712 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:57:58.713 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 53 M 2023-11-15T04:57:58.713 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 229 M 2023-11-15T04:57:58.713 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:57:58.790 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:57:58.791 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:58.791 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:57:58.792 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:58.792 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:57:58.792 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T04:57:58.792 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:57:58.792 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:58.793 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:57:58.794 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:57:58.794 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2023-11-15T04:57:58.794 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:57:58.795 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 53 M 2023-11-15T04:57:58.795 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 229 M 2023-11-15T04:57:58.796 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:57:58.948 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 169 kB 00:00 2023-11-15T04:57:59.042 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 169 kB 00:00 2023-11-15T04:57:59.565 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 78 MB/s | 52 MB 00:00 2023-11-15T04:57:59.590 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 295 kB/s | 203 kB 00:00 2023-11-15T04:57:59.624 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 277 kB/s | 187 kB 00:00 2023-11-15T04:57:59.674 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 847 kB/s | 71 kB 00:00 2023-11-15T04:57:59.734 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2023-11-15T04:57:59.760 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 263 kB/s | 203 kB 00:00 2023-11-15T04:57:59.777 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 255 kB/s | 187 kB 00:00 2023-11-15T04:57:59.844 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 846 kB/s | 71 kB 00:00 2023-11-15T04:57:59.941 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 861 kB/s | 323 kB 00:00 2023-11-15T04:57:59.942 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:57:59.942 INFO:teuthology.orchestra.run.smithi086.stdout:Total 43 MB/s | 53 MB 00:01 2023-11-15T04:57:59.985 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:00.006 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:00.007 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:00.110 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 860 kB/s | 323 kB 00:00 2023-11-15T04:58:00.111 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:00.112 INFO:teuthology.orchestra.run.smithi154.stdout:Total 40 MB/s | 53 MB 00:01 2023-11-15T04:58:00.155 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:00.177 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:00.177 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:00.309 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:00.313 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:00.486 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:00.489 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:00.784 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:00.964 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:01.035 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T04:58:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:58:01.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T04:58:01.407 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:58:01.487 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:58:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:58:01.657 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T04:58:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:58:01.812 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T04:58:01.849 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:58:01.947 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:58:06.709 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:58:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T04:58:06.946 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:58:07.053 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:58:07.237 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:58:07.237 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:58:07.238 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:58:07.238 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:58:07.238 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:58:07.238 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:58:07.347 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T04:58:07.348 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T04:58:07.348 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T04:58:07.349 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T04:58:07.349 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T04:58:07.349 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T04:58:07.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:58:07.951 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:07.952 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:58:07.952 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:58:07.952 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:58:07.952 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:58:07.952 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:07.953 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:08.031 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T04:58:08.032 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:08.032 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:58:08.032 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T04:58:08.032 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T04:58:08.033 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T04:58:08.033 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:08.033 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:08.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2023-11-15T04:58:08.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2023-11-15T04:58:09.050 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:54 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:09.128 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:10.717 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:10.720 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:10.721 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:58:10.721 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:10.721 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:58:10.721 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:58:10.721 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:58:10.722 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:58:10.722 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:58:10.722 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:58:10.722 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:58:10.723 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:58:10.723 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:58:10.723 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:58:10.723 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:58:10.723 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:58:10.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:58:10.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:58:10.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:58:10.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:58:10.724 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:58:10.725 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:58:10.725 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:58:10.725 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:58:10.725 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:58:10.725 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:58:10.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:58:10.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:58:10.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:58:10.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:58:10.727 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:58:10.727 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:58:10.727 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:58:10.727 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:58:10.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:58:10.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:58:10.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:58:10.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:58:10.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:10.729 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:58:10.729 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:10.729 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2023-11-15T04:58:10.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:10.730 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 M 2023-11-15T04:58:10.730 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 111 M 2023-11-15T04:58:10.731 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:58:10.800 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:10.803 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:58:10.804 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:10.804 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:58:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T04:58:10.804 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:58:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T04:58:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T04:58:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T04:58:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T04:58:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T04:58:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T04:58:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T04:58:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T04:58:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T04:58:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T04:58:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T04:58:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T04:58:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T04:58:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T04:58:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T04:58:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T04:58:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T04:58:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T04:58:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T04:58:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T04:58:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T04:58:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T04:58:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T04:58:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T04:58:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T04:58:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T04:58:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:10.811 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:58:10.811 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:10.812 INFO:teuthology.orchestra.run.smithi154.stdout:Install 32 Packages 2023-11-15T04:58:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:10.812 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 31 M 2023-11-15T04:58:10.813 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 111 M 2023-11-15T04:58:10.813 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:58:11.403 INFO:teuthology.orchestra.run.smithi154.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 139 kB/s | 6.3 kB 00:00 2023-11-15T04:58:11.453 INFO:teuthology.orchestra.run.smithi154.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-15T04:58:11.504 INFO:teuthology.orchestra.run.smithi154.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-11-15T04:58:11.529 INFO:teuthology.orchestra.run.smithi154.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T04:58:11.613 INFO:teuthology.orchestra.run.smithi154.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2023-11-15T04:58:11.808 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 121 kB/s | 6.3 kB 00:00 2023-11-15T04:58:11.867 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2023-11-15T04:58:11.880 INFO:teuthology.orchestra.run.smithi154.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2023-11-15T04:58:11.905 INFO:teuthology.orchestra.run.smithi154.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2023-11-15T04:58:11.917 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-15T04:58:11.930 INFO:teuthology.orchestra.run.smithi154.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 392 kB/s | 157 kB 00:00 2023-11-15T04:58:11.943 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-11-15T04:58:12.026 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2023-11-15T04:58:12.030 INFO:teuthology.orchestra.run.smithi154.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 456 kB/s | 57 kB 00:00 2023-11-15T04:58:12.106 INFO:teuthology.orchestra.run.smithi154.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2023-11-15T04:58:12.148 INFO:teuthology.orchestra.run.smithi154.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 387 kB/s | 103 kB 00:00 2023-11-15T04:58:12.173 INFO:teuthology.orchestra.run.smithi154.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 249 kB 00:00 2023-11-15T04:58:12.256 INFO:teuthology.orchestra.run.smithi154.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 4.2 MB/s | 641 kB 00:00 2023-11-15T04:58:12.282 INFO:teuthology.orchestra.run.smithi154.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 814 kB/s | 109 kB 00:00 2023-11-15T04:58:12.285 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2023-11-15T04:58:12.310 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2023-11-15T04:58:12.336 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2023-11-15T04:58:12.444 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 427 kB/s | 57 kB 00:00 2023-11-15T04:58:12.520 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.4 MB/s | 457 kB 00:00 2023-11-15T04:58:12.553 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 387 kB/s | 103 kB 00:00 2023-11-15T04:58:12.587 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 249 kB 00:00 2023-11-15T04:58:12.679 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 3.9 MB/s | 641 kB 00:00 2023-11-15T04:58:12.704 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 725 kB/s | 109 kB 00:00 2023-11-15T04:58:12.746 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-11-15T04:58:12.815 INFO:teuthology.orchestra.run.smithi154.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 68 kB/s | 44 kB 00:00 2023-11-15T04:58:12.913 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 792 kB/s | 185 kB 00:00 2023-11-15T04:58:12.938 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2023-11-15T04:58:12.965 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 666 kB/s | 173 kB 00:00 2023-11-15T04:58:12.989 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-11-15T04:58:13.005 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-11-15T04:58:13.023 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 827 kB/s | 49 kB 00:00 2023-11-15T04:58:13.048 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 997 kB/s | 59 kB 00:00 2023-11-15T04:58:13.082 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-11-15T04:58:13.098 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2023-11-15T04:58:13.115 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 279 kB/s | 19 kB 00:00 2023-11-15T04:58:13.132 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 474 kB/s | 24 kB 00:00 2023-11-15T04:58:13.149 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 577 kB/s | 29 kB 00:00 2023-11-15T04:58:13.166 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 546 kB/s | 27 kB 00:00 2023-11-15T04:58:13.208 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T04:58:13.234 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2023-11-15T04:58:13.257 INFO:teuthology.orchestra.run.smithi154.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 871 kB/s | 384 kB 00:00 2023-11-15T04:58:13.282 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.9 MB/s | 251 kB 00:00 2023-11-15T04:58:13.309 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-11-15T04:58:13.310 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:13.311 INFO:teuthology.orchestra.run.smithi086.stdout:Total 12 MB/s | 31 MB 00:02 2023-11-15T04:58:13.357 INFO:teuthology.orchestra.run.smithi154.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-11-15T04:58:13.383 INFO:teuthology.orchestra.run.smithi154.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 157 kB/s | 173 kB 00:01 2023-11-15T04:58:13.395 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:13.433 INFO:teuthology.orchestra.run.smithi154.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 980 kB/s | 49 kB 00:00 2023-11-15T04:58:13.434 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:13.434 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:13.458 INFO:teuthology.orchestra.run.smithi154.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-11-15T04:58:13.484 INFO:teuthology.orchestra.run.smithi154.stdout:(21/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 151 kB/s | 185 kB 00:01 2023-11-15T04:58:13.509 INFO:teuthology.orchestra.run.smithi154.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 775 kB/s | 59 kB 00:00 2023-11-15T04:58:13.534 INFO:teuthology.orchestra.run.smithi154.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-11-15T04:58:13.560 INFO:teuthology.orchestra.run.smithi154.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2023-11-15T04:58:13.585 INFO:teuthology.orchestra.run.smithi154.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2023-11-15T04:58:13.602 INFO:teuthology.orchestra.run.smithi154.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 357 kB/s | 24 kB 00:00 2023-11-15T04:58:13.627 INFO:teuthology.orchestra.run.smithi154.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 436 kB/s | 29 kB 00:00 2023-11-15T04:58:13.661 INFO:teuthology.orchestra.run.smithi154.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-11-15T04:58:13.711 INFO:teuthology.orchestra.run.smithi154.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-11-15T04:58:13.736 INFO:teuthology.orchestra.run.smithi154.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2023-11-15T04:58:13.750 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:13.753 INFO:teuthology.orchestra.run.smithi154.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 547 kB/s | 23 kB 00:00 2023-11-15T04:58:13.753 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:13.820 INFO:teuthology.orchestra.run.smithi154.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2023-11-15T04:58:13.821 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:13.822 INFO:teuthology.orchestra.run.smithi154.stdout:Total 10 MB/s | 31 MB 00:03 2023-11-15T04:58:13.907 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:13.945 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:13.946 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:14.253 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:14.256 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:14.860 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:15.076 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:58:15.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:58:15.386 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:15.555 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:58:15.575 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T04:58:15.761 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:58:15.786 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T04:58:15.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:58:16.019 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T04:58:16.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:58:16.217 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T04:58:16.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:58:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T04:58:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:58:16.603 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T04:58:16.663 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:58:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T04:58:16.882 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:58:16.958 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T04:58:17.124 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:58:17.176 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T04:58:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:58:17.372 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T04:58:17.592 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T04:58:17.716 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:58:17.892 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T04:58:17.974 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:58:18.154 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:58:18.259 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T04:58:18.396 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:58:18.489 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T04:58:18.581 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:58:18.660 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T04:58:18.859 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:58:18.902 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T04:58:19.069 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:58:19.078 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:58:19.250 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:58:19.355 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T04:58:19.483 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:58:19.583 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T04:58:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:58:19.779 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T04:58:19.974 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T04:58:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:58:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T04:58:20.212 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:58:20.432 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T04:58:20.575 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:58:20.663 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T04:58:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:58:21.033 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T04:58:21.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:58:21.301 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T04:58:21.496 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:58:21.695 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T04:58:21.890 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:58:23.030 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:58:23.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:58:23.422 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T04:58:23.598 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:58:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:58:23.896 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:58:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T04:58:24.118 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:58:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:58:24.313 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:58:24.331 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:58:24.470 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:58:24.471 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T04:58:24.675 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:58:24.693 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T04:58:24.826 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:58:24.896 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:58:24.897 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:58:24.897 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:58:24.897 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:58:24.897 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:58:24.898 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:58:24.898 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:58:24.898 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:58:24.899 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:58:24.899 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:58:24.899 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:58:24.899 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:58:24.899 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:58:24.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:58:24.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:58:24.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:58:24.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:58:24.900 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:58:24.901 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:58:24.901 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:58:24.901 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:58:24.901 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:58:24.901 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:58:24.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:58:24.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:58:24.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:58:24.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:58:24.902 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:58:24.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:58:24.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:58:24.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:58:24.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:58:25.234 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T04:58:25.235 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T04:58:25.235 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T04:58:25.235 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T04:58:25.235 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T04:58:25.235 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T04:58:25.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T04:58:25.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T04:58:25.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T04:58:25.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T04:58:25.236 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T04:58:25.237 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T04:58:25.237 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T04:58:25.237 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T04:58:25.237 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T04:58:25.237 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T04:58:25.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T04:58:25.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T04:58:25.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T04:58:25.238 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T04:58:25.239 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T04:58:25.239 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T04:58:25.239 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T04:58:25.239 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T04:58:25.239 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T04:58:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T04:58:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T04:58:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T04:58:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T04:58:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T04:58:25.241 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T04:58:25.242 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T04:58:26.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:58:26.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:26.038 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:58:26.038 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.038 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.038 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.038 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:58:26.039 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.039 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.039 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:58:26.039 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:58:26.039 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:58:26.040 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:58:26.040 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:58:26.040 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:58:26.040 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:58:26.040 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:58:26.041 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:58:26.041 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:58:26.041 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:58:26.041 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:58:26.041 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:58:26.042 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:58:26.042 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:58:26.042 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:58:26.042 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:58:26.042 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:58:26.043 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:58:26.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:58:26.051 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:58:26.051 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:58:26.051 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:58:26.051 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:58:26.051 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:58:26.052 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:58:26.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:26.052 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:26.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2023-11-15T04:58:26.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T04:58:26.441 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:26.441 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:58:26.441 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.441 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.442 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.442 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T04:58:26.442 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.442 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:26.442 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T04:58:26.443 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T04:58:26.443 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T04:58:26.443 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T04:58:26.443 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T04:58:26.443 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T04:58:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T04:58:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T04:58:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T04:58:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T04:58:26.444 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T04:58:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T04:58:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T04:58:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T04:58:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T04:58:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T04:58:26.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T04:58:26.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T04:58:26.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T04:58:26.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T04:58:26.447 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T04:58:26.448 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T04:58:26.448 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T04:58:26.448 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T04:58:26.449 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T04:58:26.449 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T04:58:26.449 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:26.449 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:26.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2023-11-15T04:58:27.117 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:27.456 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:28.583 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:58:28.798 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:28.799 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:58:28.799 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:28.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2023-11-15T04:58:28.907 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:58:29.122 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:29.124 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:58:29.124 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:29.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2023-11-15T04:58:29.754 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:30.112 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:31.430 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:31.432 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:31.432 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:58:31.433 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:31.433 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:58:31.433 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:58:31.434 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:31.434 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:58:31.435 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:31.435 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:58:31.435 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:31.435 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 71 k 2023-11-15T04:58:31.435 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 290 k 2023-11-15T04:58:31.436 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:58:31.476 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2023-11-15T04:58:31.477 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:31.477 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-11-15T04:58:31.477 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:31.483 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:31.484 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:31.485 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:31.489 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:31.535 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:31.769 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:31.770 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:31.770 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:58:31.770 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:31.770 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:58:31.771 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T04:58:31.771 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:31.771 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:58:31.771 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:31.771 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:58:31.772 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:31.772 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 71 k 2023-11-15T04:58:31.772 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 290 k 2023-11-15T04:58:31.772 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:58:31.817 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-11-15T04:58:31.818 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:31.818 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-11-15T04:58:31.818 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:31.825 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:31.825 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:31.827 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:31.829 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:31.873 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:31.985 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:32.207 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:32.389 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:32.427 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:32.593 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:33.064 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:33.065 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:58:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:58:33.065 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:33.065 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:33.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T04:58:33.485 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:33.486 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:58:33.486 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T04:58:33.486 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:33.486 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:33.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:58:34.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T04:58:34.764 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:35.403 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:20 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:36.419 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:36.420 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-11-15T04:58:36.420 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:36.421 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:58:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:58:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:36.421 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:58:36.421 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:36.422 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:58:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:36.422 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 163 k 2023-11-15T04:58:36.422 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 587 k 2023-11-15T04:58:36.422 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:58:36.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2023-11-15T04:58:36.465 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:36.466 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.6 MB/s | 163 kB 00:00 2023-11-15T04:58:36.466 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:36.476 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:36.476 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:36.522 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:36.525 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:36.631 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:36.819 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:37.066 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:37.067 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:37.067 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2023-11-15T04:58:37.067 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:37.068 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:58:37.068 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T04:58:37.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:37.068 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:58:37.069 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:37.069 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:58:37.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:37.069 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 163 k 2023-11-15T04:58:37.070 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 587 k 2023-11-15T04:58:37.070 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:58:37.115 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-11-15T04:58:37.116 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:37.117 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-11-15T04:58:37.117 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:37.127 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:37.127 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:37.173 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:37.176 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:37.252 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:37.287 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:37.883 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:37.932 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:37.932 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:58:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:37.933 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:37.933 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:38.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2023-11-15T04:58:38.554 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:58:38.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:38.555 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:58:38.555 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T04:58:38.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:38.555 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:38.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2023-11-15T04:58:39.018 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:39.626 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:40.453 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:58:40.671 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:40.672 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:58:40.673 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:40.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:58:41.186 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:58:41.405 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:41.407 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:58:41.407 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:41.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T04:58:43.299 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:43.300 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:43.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:43.300 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:58:43.301 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:43.301 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:58:43.301 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:58:43.301 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:58:43.301 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:58:43.302 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:58:43.302 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:58:43.302 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:58:43.302 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:58:43.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:43.303 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:58:43.303 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:58:43.303 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-11-15T04:58:43.303 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:43.303 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.0 M 2023-11-15T04:58:43.304 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 15 M 2023-11-15T04:58:43.304 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:58:43.601 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 404 kB/s | 24 kB 00:00 2023-11-15T04:58:43.626 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 106 kB/s | 8.9 kB 00:00 2023-11-15T04:58:43.693 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2023-11-15T04:58:43.768 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2023-11-15T04:58:43.794 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2023-11-15T04:58:43.902 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 942 kB/s | 196 kB 00:00 2023-11-15T04:58:43.904 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:43.905 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.8 MB/s | 3.0 MB 00:00 2023-11-15T04:58:43.914 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:43.924 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:43.924 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:44.081 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:44.084 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:44.176 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:44.598 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:58:44.852 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:58:45.053 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:58:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:58:45.850 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:45.851 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:58:45.851 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:45.851 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:58:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T04:58:45.852 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:58:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T04:58:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T04:58:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T04:58:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T04:58:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T04:58:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:45.853 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:58:45.853 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:58:45.853 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2023-11-15T04:58:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:45.854 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.0 M 2023-11-15T04:58:45.855 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 15 M 2023-11-15T04:58:45.855 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:58:46.093 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 434 kB/s | 24 kB 00:00 2023-11-15T04:58:46.118 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 111 kB/s | 8.9 kB 00:00 2023-11-15T04:58:46.185 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-11-15T04:58:46.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:58:46.261 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2023-11-15T04:58:46.369 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2023-11-15T04:58:46.459 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:58:46.577 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 501 kB/s | 196 kB 00:00 2023-11-15T04:58:46.579 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:46.579 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.1 MB/s | 3.0 MB 00:00 2023-11-15T04:58:46.590 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:46.600 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:46.601 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:58:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:58:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:58:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:58:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:58:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:58:46.765 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:46.768 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:47.030 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:47.236 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T04:58:47.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:58:47.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:47.404 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:58:47.404 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:58:47.405 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:58:47.405 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:58:47.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:58:47.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:58:47.405 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:58:47.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:47.406 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:58:47.523 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T04:58:47.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:58:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T04:58:47.908 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T04:58:48.385 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:48.889 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T04:58:49.135 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:58:49.439 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:58:49.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:58:49.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T04:58:49.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T04:58:49.440 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T04:58:49.441 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T04:58:50.044 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:58:50.045 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-11-15T04:58:50.046 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:58:50.046 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-11-15T04:58:50.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:58:50.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:58:50.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:58:50.048 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:58:50.048 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:58:50.048 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:58:50.049 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:58:50.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:58:50.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:58:50.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:58:50.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:50.051 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:58:50.052 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-11-15T04:58:50.052 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2023-11-15T04:58:50.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:58:50.054 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2023-11-15T04:58:50.055 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 210 M 2023-11-15T04:58:50.055 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:58:50.102 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T04:58:50.103 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:50.103 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:58:50.103 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T04:58:50.103 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T04:58:50.104 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T04:58:50.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T04:58:50.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T04:58:50.104 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T04:58:50.104 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:50.105 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:58:50.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T04:58:50.353 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 37 MB/s | 8.6 MB 00:00 2023-11-15T04:58:50.710 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2023-11-15T04:58:50.785 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-11-15T04:58:51.088 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:58:51.110 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.9 MB 00:00 2023-11-15T04:58:51.178 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.5 MB/s | 3.7 MB 00:00 2023-11-15T04:58:51.369 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 23 MB/s | 14 MB 00:00 2023-11-15T04:58:51.394 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 796 kB/s | 172 kB 00:00 2023-11-15T04:58:51.445 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-11-15T04:58:51.445 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:51.446 INFO:teuthology.orchestra.run.smithi086.stdout:Total 26 MB/s | 37 MB 00:01 2023-11-15T04:58:51.675 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:58:51.693 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:58:51.693 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:58:52.012 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:58:52.015 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:58:52.653 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:58:52.761 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:58:52.762 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================ 2023-11-15T04:58:52.762 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:58:52.762 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================ 2023-11-15T04:58:52.762 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:58:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T04:58:52.763 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:58:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T04:58:52.763 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T04:58:52.764 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T04:58:52.764 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T04:58:52.764 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T04:58:52.764 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T04:58:52.765 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T04:58:52.765 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:52.765 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:58:52.766 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================================ 2023-11-15T04:58:52.766 INFO:teuthology.orchestra.run.smithi154.stdout:Install 8 Packages 2023-11-15T04:58:52.767 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:58:52.768 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 37 M 2023-11-15T04:58:52.768 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 210 M 2023-11-15T04:58:52.768 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:58:52.784 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:58:52.916 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:58:52.976 INFO:teuthology.orchestra.run.smithi154.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 63 MB/s | 8.6 MB 00:00 2023-11-15T04:58:53.084 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:58:53.501 INFO:teuthology.orchestra.run.smithi154.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 7.1 MB/s | 4.7 MB 00:00 2023-11-15T04:58:53.568 INFO:teuthology.orchestra.run.smithi154.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.3 MB/s | 3.7 MB 00:00 2023-11-15T04:58:53.593 INFO:teuthology.orchestra.run.smithi154.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-15T04:58:53.675 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:58:53.702 INFO:teuthology.orchestra.run.smithi154.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.6 MB/s | 4.9 MB 00:00 2023-11-15T04:58:53.802 INFO:teuthology.orchestra.run.smithi154.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2023-11-15T04:58:53.844 INFO:teuthology.orchestra.run.smithi154.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2023-11-15T04:58:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:58:54.053 INFO:teuthology.orchestra.run.smithi154.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2023-11-15T04:58:54.053 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:58:54.054 INFO:teuthology.orchestra.run.smithi154.stdout:Total 28 MB/s | 37 MB 00:01 2023-11-15T04:58:54.286 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:58:54.304 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:58:54.305 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:58:54.470 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:58:54.625 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:58:54.626 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:58:54.628 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:58:55.220 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:58:55.289 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:58:55.371 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:58:55.509 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T04:58:55.627 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:58:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:58:56.087 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:58:56.808 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T04:58:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:58:57.648 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:58:57.821 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:58:58.308 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:58:58.510 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T04:58:58.858 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:58:59.104 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:58:59.637 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:58:59.824 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:59:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:59:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:59:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:59:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:59:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:59:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:59:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:59:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:59:00.834 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:59:00.835 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:00.835 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:59:00.836 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:59:00.836 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:59:00.836 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:59:00.836 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:59:00.836 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:59:00.837 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:59:00.837 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:59:00.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T04:59:00.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:59:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:02.182 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T04:59:02.404 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:59:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T04:59:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T04:59:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T04:59:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T04:59:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T04:59:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T04:59:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T04:59:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T04:59:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T04:59:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:03.398 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T04:59:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T04:59:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T04:59:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T04:59:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T04:59:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T04:59:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T04:59:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T04:59:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:03.400 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:03.437 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:59:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:03.438 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:59:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:59:03.438 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:59:03.438 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:59:03.439 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:59:03.439 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:59:03.439 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:59:03.439 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:59:03.439 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:59:03.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:59:03.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:59:03.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:59:03.440 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:59:03.441 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:59:03.441 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2023-11-15T04:59:03.442 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2023-11-15T04:59:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2023-11-15T04:59:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:03.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2023-11-15T04:59:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 579 kB/s | 27 kB 00:00 2023-11-15T04:59:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 148 kB/s | 27 kB 00:00 2023-11-15T04:59:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 97 kB/s | 20 kB 00:00 2023-11-15T04:59:04.378 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:49 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-11-15T04:59:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2023-11-15T04:59:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 625 kB/s | 110 kB 00:00 2023-11-15T04:59:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 562 kB/s | 43 kB 00:00 2023-11-15T04:59:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2023-11-15T04:59:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 150 kB/s | 16 kB 00:00 2023-11-15T04:59:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 608 kB/s | 81 kB 00:00 2023-11-15T04:59:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-11-15T04:59:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-11-15T04:59:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.0 MB/s | 1.2 MB 00:00 2023-11-15T04:59:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-11-15T04:59:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:05.409 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:05.662 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:59:05.875 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:59:06.065 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:06.070 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:06.070 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:59:06.070 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:06.070 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:06.071 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T04:59:06.071 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:59:06.071 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T04:59:06.071 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T04:59:06.072 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T04:59:06.072 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T04:59:06.072 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T04:59:06.072 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T04:59:06.073 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T04:59:06.073 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T04:59:06.073 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T04:59:06.073 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T04:59:06.073 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T04:59:06.074 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T04:59:06.074 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:06.074 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:06.074 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:06.075 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2023-11-15T04:59:06.075 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:06.075 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.9 M 2023-11-15T04:59:06.075 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2023-11-15T04:59:06.076 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:59:06.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:59:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:59:06.559 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 387 kB/s | 27 kB 00:00 2023-11-15T04:59:06.693 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 98 kB/s | 20 kB 00:00 2023-11-15T04:59:06.718 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 116 kB/s | 27 kB 00:00 2023-11-15T04:59:06.804 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:59:06.827 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-11-15T04:59:06.844 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2023-11-15T04:59:06.861 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-11-15T04:59:06.886 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 570 kB/s | 110 kB 00:00 2023-11-15T04:59:06.978 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 139 kB/s | 16 kB 00:00 2023-11-15T04:59:06.995 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 218 kB/s | 33 kB 00:00 2023-11-15T04:59:07.045 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2023-11-15T04:59:07.071 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:59:07.087 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 698 kB/s | 64 kB 00:00 2023-11-15T04:59:07.104 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-11-15T04:59:07.196 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.4 MB/s | 1.2 MB 00:00 2023-11-15T04:59:07.196 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:07.197 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-11-15T04:59:07.255 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:07.273 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:07.275 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:59:07.356 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:07.359 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:07.880 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:08.121 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T04:59:08.319 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T04:59:08.337 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:59:08.571 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T04:59:08.647 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:59:08.784 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T04:59:08.855 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:59:08.973 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T04:59:09.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:59:09.198 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:59:09.205 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T04:59:09.480 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T04:59:09.499 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:59:09.499 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:59:09.499 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:59:09.500 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:59:09.501 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:59:09.501 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:59:09.501 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:59:09.501 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:59:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T04:59:10.227 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:59:10.228 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:10.228 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:10.228 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:59:10.228 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:59:10.229 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:59:10.229 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:59:10.229 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:59:10.229 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:59:10.229 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:59:10.230 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:59:10.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:10.231 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:10.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:59:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T04:59:11.092 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T04:59:11.191 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:56 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:11.275 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T04:59:11.451 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T04:59:11.648 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:59:11.951 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T04:59:11.952 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T04:59:11.952 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T04:59:11.952 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T04:59:11.952 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T04:59:11.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T04:59:11.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T04:59:11.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T04:59:11.953 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T04:59:11.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T04:59:11.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T04:59:11.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T04:59:11.954 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T04:59:12.694 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T04:59:12.694 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:12.694 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:12.695 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T04:59:12.695 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T04:59:12.695 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T04:59:12.695 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T04:59:12.695 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T04:59:12.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T04:59:12.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T04:59:12.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T04:59:12.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T04:59:12.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T04:59:12.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T04:59:12.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T04:59:12.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T04:59:12.697 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:12.697 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:12.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T04:59:12.862 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:12.863 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:12.863 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-11-15T04:59:12.863 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:12.864 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:12.864 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:59:12.864 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:59:12.864 INFO:teuthology.orchestra.run.smithi086.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:59:12.864 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:59:12.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:59:12.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:59:12.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:59:12.865 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:12.866 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:12.866 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:12.866 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-11-15T04:59:12.866 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:12.867 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.8 M 2023-11-15T04:59:12.867 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2023-11-15T04:59:12.867 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:13.005 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2023-11-15T04:59:13.055 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 423 kB/s | 21 kB 00:00 2023-11-15T04:59:13.122 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 5.2 MB/s | 355 kB 00:00 2023-11-15T04:59:13.156 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.4 MB/s | 49 kB 00:00 2023-11-15T04:59:13.339 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-11-15T04:59:13.548 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.2 MB/s | 4.8 MB 00:00 2023-11-15T04:59:13.549 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:13.549 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.5 MB/s | 5.8 MB 00:00 2023-11-15T04:59:13.600 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:13.609 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:13.609 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:13.678 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:58 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:13.718 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:13.722 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:14.006 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:14.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:59:14.594 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:59:15.339 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:15.340 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:15.340 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2023-11-15T04:59:15.340 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:15.341 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:15.341 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T04:59:15.341 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:59:15.341 INFO:teuthology.orchestra.run.smithi154.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T04:59:15.342 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T04:59:15.342 INFO:teuthology.orchestra.run.smithi154.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T04:59:15.342 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T04:59:15.342 INFO:teuthology.orchestra.run.smithi154.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T04:59:15.342 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:15.343 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:15.343 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:15.343 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2023-11-15T04:59:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:15.344 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 5.8 M 2023-11-15T04:59:15.344 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2023-11-15T04:59:15.344 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:15.544 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:59:15.745 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2023-11-15T04:59:15.927 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:59:16.137 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 54 kB/s | 21 kB 00:00 2023-11-15T04:59:16.162 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.1 MB/s | 538 kB 00:00 2023-11-15T04:59:16.168 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:59:16.382 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:59:16.429 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.6 MB/s | 4.8 MB 00:00 2023-11-15T04:59:16.546 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 870 kB/s | 355 kB 00:00 2023-11-15T04:59:16.571 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 119 kB/s | 49 kB 00:00 2023-11-15T04:59:16.573 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:16.574 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.7 MB/s | 5.8 MB 00:01 2023-11-15T04:59:16.624 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:16.634 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:16.634 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:16.655 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:59:16.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:59:16.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:59:16.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:59:16.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:59:16.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:59:16.734 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:16.737 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:17.042 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:17.266 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T04:59:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:59:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:17.331 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:59:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:59:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:59:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:17.332 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:17.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2023-11-15T04:59:17.517 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T04:59:18.310 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:18.499 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T04:59:18.863 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T04:59:19.071 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T04:59:19.267 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:59:19.571 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T04:59:19.571 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T04:59:19.572 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T04:59:19.572 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T04:59:19.572 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T04:59:19.572 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T04:59:19.974 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:19.974 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:19.975 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:19.975 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:59:19.975 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-11-15T04:59:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:59:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:19.976 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:19.976 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:19.976 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2023-11-15T04:59:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:19.977 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 968 k 2023-11-15T04:59:19.977 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.2 M 2023-11-15T04:59:19.978 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:20.076 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2023-11-15T04:59:20.143 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 644 kB/s | 83 kB 00:00 2023-11-15T04:59:20.144 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:20.144 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.6 MB/s | 968 kB 00:00 2023-11-15T04:59:20.154 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:20.174 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:20.175 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T04:59:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:20.247 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T04:59:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T04:59:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T04:59:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:20.248 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:20.261 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:20.264 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:20.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2023-11-15T04:59:20.438 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:20.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:59:20.897 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:59:21.244 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:06 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:21.304 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:59:21.304 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:59:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:59:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:21.984 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:59:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:21.985 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:22.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2023-11-15T04:59:22.914 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:22.915 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:22.915 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:22.915 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:22.916 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:22.916 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T04:59:22.916 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2023-11-15T04:59:22.916 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T04:59:22.916 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:22.917 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:22.917 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:22.917 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2023-11-15T04:59:22.917 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:22.918 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 968 k 2023-11-15T04:59:22.918 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 3.2 M 2023-11-15T04:59:22.919 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:23.074 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:23.184 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 11 MB/s | 885 kB 00:00 2023-11-15T04:59:23.258 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 549 kB/s | 83 kB 00:00 2023-11-15T04:59:23.258 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:23.259 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.8 MB/s | 968 kB 00:00 2023-11-15T04:59:23.269 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:23.289 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:23.290 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:23.376 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:23.379 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:23.548 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:23.808 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T04:59:23.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:59:24.364 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T04:59:24.364 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T04:59:24.735 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:24.736 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:24.737 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:24.738 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 131 k 2023-11-15T04:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 439 k 2023-11-15T04:59:24.739 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:24.797 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 131 kB 00:00 2023-11-15T04:59:24.798 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:24.798 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.1 MB/s | 131 kB 00:00 2023-11-15T04:59:24.799 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:24.804 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:24.804 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:24.853 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:24.856 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:24.973 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T04:59:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:25.059 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T04:59:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:25.060 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:25.212 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:25.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2023-11-15T04:59:25.498 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:26.163 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:26.177 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:26.178 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:26.178 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:26.178 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:26.178 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:26.179 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:26.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2023-11-15T04:59:27.276 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:27.824 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:27.825 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:27.826 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:27.826 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:27.826 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:27.826 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T04:59:27.827 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:27.827 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:27.827 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:27.827 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:59:27.828 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:27.828 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 131 k 2023-11-15T04:59:27.828 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 439 k 2023-11-15T04:59:27.828 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:27.870 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 131 kB 00:00 2023-11-15T04:59:27.871 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:27.871 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.9 MB/s | 131 kB 00:00 2023-11-15T04:59:27.871 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:27.876 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:27.877 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:27.925 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:27.928 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:28.456 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:28.712 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:28.928 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:28.929 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:28.929 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:29.090 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:29.398 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:29.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2023-11-15T04:59:30.044 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:30.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:30.045 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:30.045 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:30.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:30.046 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:30.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2023-11-15T04:59:30.496 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:31.118 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:16 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:32.169 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:32.169 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:32.170 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:32.170 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:32.170 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:32.170 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:59:32.170 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:32.171 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:32.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:32.171 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:59:32.171 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:32.172 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 24 k 2023-11-15T04:59:32.172 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 94 k 2023-11-15T04:59:32.172 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:32.212 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 582 kB/s | 24 kB 00:00 2023-11-15T04:59:32.213 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:32.213 INFO:teuthology.orchestra.run.smithi086.stdout:Total 565 kB/s | 24 kB 00:00 2023-11-15T04:59:32.214 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:32.216 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:32.217 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:32.219 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:32.222 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:32.271 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:32.485 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:32.568 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:32.772 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:32.783 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:32.785 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:32.785 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:32.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2023-11-15T04:59:33.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:33.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:33.399 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:33.399 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:33.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:33.400 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:33.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2023-11-15T04:59:33.747 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:35.405 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:35.406 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:35.406 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:35.406 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:35.407 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T04:59:35.407 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:35.407 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:35.407 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:35.408 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:59:35.408 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:35.408 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 24 k 2023-11-15T04:59:35.408 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 94 k 2023-11-15T04:59:35.408 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:35.451 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 547 kB/s | 24 kB 00:00 2023-11-15T04:59:35.452 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:35.452 INFO:teuthology.orchestra.run.smithi154.stdout:Total 531 kB/s | 24 kB 00:00 2023-11-15T04:59:35.453 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:35.456 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:35.457 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:35.458 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:35.461 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:35.513 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:35.679 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:35.833 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:35.940 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:36.047 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:36.048 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:36.049 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:36.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2023-11-15T04:59:36.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:36.595 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:36.596 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:36.596 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:36.596 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:36.596 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:36.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2023-11-15T04:59:37.010 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:37.591 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:22 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:38.467 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:38.682 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:38.685 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:38.685 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:38.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2023-11-15T04:59:39.032 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:39.247 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:39.248 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:39.248 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:39.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2023-11-15T04:59:39.663 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:40.232 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:41.142 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:41.358 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:41.359 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:41.359 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:41.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2023-11-15T04:59:41.684 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:41.898 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:41.899 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:41.899 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:42.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2023-11-15T04:59:42.319 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:42.880 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:43.795 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:44.010 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:44.011 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:44.011 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:44.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2023-11-15T04:59:44.357 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:44.572 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:44.573 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:44.574 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:44.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2023-11-15T04:59:44.974 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:45.554 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:46.420 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:46.638 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:46.639 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:46.639 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:46.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2023-11-15T04:59:47.014 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:47.229 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:47.230 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:47.230 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:47.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2023-11-15T04:59:47.594 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:48.204 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:49.043 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:49.262 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:49.263 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-11-15T04:59:49.263 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:49.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2023-11-15T04:59:49.657 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:49.872 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:49.873 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:49.873 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:49.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2023-11-15T04:59:50.242 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:50.841 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:51.892 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:51.893 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:51.893 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:51.893 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:51.893 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:51.893 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:59:51.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:51.894 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:51.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:51.894 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:59:51.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:51.895 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 89 k 2023-11-15T04:59:51.895 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 298 k 2023-11-15T04:59:51.895 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:51.936 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2023-11-15T04:59:51.937 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:51.937 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.0 MB/s | 89 kB 00:00 2023-11-15T04:59:51.937 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:51.946 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:51.949 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:51.996 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:51.998 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:52.267 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T04:59:52.333 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:52.483 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:52.484 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2023-11-15T04:59:52.485 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:52.613 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:52.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2023-11-15T04:59:53.276 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:53.276 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:53.277 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:53.277 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:53.277 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:53.277 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:53.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2023-11-15T04:59:53.491 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:38 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:54.313 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:55.161 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:55.161 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:55.161 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T04:59:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:55.162 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:55.162 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:55.162 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:59:55.162 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:55.163 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 89 k 2023-11-15T04:59:55.163 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 298 k 2023-11-15T04:59:55.163 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:55.207 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2023-11-15T04:59:55.208 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:55.208 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.9 MB/s | 89 kB 00:00 2023-11-15T04:59:55.208 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:55.218 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:55.218 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:55.266 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:55.269 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:55.377 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T04:59:55.559 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:55.964 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T04:59:55.965 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:55.965 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:55.965 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:55.965 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T04:59:55.965 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:59:55.966 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:55.966 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T04:59:55.966 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T04:59:55.966 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T04:59:55.966 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:55.967 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2023-11-15T04:59:55.967 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 18 M 2023-11-15T04:59:55.968 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T04:59:56.064 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2023-11-15T04:59:56.065 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:56.065 INFO:teuthology.orchestra.run.smithi086.stdout:Total 37 MB/s | 3.7 MB 00:00 2023-11-15T04:59:56.065 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T04:59:56.077 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T04:59:56.077 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T04:59:56.139 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T04:59:56.142 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T04:59:56.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:56.578 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:56.578 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T04:59:56.578 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:56.578 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:56.578 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T04:59:56.605 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T04:59:56.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2023-11-15T04:59:56.817 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:57.228 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:57.662 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:57.890 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T04:59:57.891 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:57.891 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T04:59:57.891 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T04:59:57.891 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T04:59:57.892 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T04:59:58.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2023-11-15T04:59:58.947 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:43 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T04:59:59.344 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T04:59:59.345 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:59.345 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T04:59:59.345 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:59.346 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T04:59:59.346 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T04:59:59.346 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:59.346 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T04:59:59.347 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T04:59:59.347 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T04:59:59.347 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T04:59:59.347 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.7 M 2023-11-15T04:59:59.347 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 18 M 2023-11-15T04:59:59.348 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T04:59:59.435 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2023-11-15T04:59:59.436 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T04:59:59.436 INFO:teuthology.orchestra.run.smithi154.stdout:Total 41 MB/s | 3.7 MB 00:00 2023-11-15T04:59:59.436 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T04:59:59.448 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T04:59:59.448 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T04:59:59.512 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T04:59:59.515 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T04:59:59.989 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T05:00:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-11-15T05:00:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T05:00:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-11-15T05:00:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T05:00:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-11-15T05:00:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T05:00:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T05:00:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-11-15T05:00:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-11-15T05:00:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-11-15T05:00:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T05:00:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 173 k 2023-11-15T05:00:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 649 k 2023-11-15T05:00:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-11-15T05:00:00.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 173 kB 00:00 2023-11-15T05:00:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:00:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.1 MB/s | 173 kB 00:00 2023-11-15T05:00:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-11-15T05:00:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-11-15T05:00:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-11-15T05:00:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-11-15T05:00:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-11-15T05:00:00.822 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-11-15T05:00:01.024 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:01.335 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:01.335 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:01.336 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T05:00:01.336 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T05:00:01.336 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:01.336 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T05:00:01.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2023-11-15T05:00:01.989 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:01.989 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T05:00:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-11-15T05:00:01.990 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:00:01.990 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-11-15T05:00:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-11-15T05:00:02.178 DEBUG:teuthology.parallel:result is None 2023-11-15T05:00:02.406 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:47 ago on Wed 15 Nov 2023 04:57:15 AM UTC. 2023-11-15T05:00:04.082 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2023-11-15T05:00:04.083 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T05:00:04.084 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2023-11-15T05:00:04.084 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T05:00:04.084 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2023-11-15T05:00:04.084 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T05:00:04.085 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:04.085 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2023-11-15T05:00:04.085 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2023-11-15T05:00:04.085 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2023-11-15T05:00:04.085 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:04.086 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 173 k 2023-11-15T05:00:04.086 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 649 k 2023-11-15T05:00:04.086 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2023-11-15T05:00:04.154 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 173 kB 00:00 2023-11-15T05:00:04.155 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:00:04.155 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.4 MB/s | 173 kB 00:00 2023-11-15T05:00:04.156 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2023-11-15T05:00:04.166 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2023-11-15T05:00:04.166 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2023-11-15T05:00:04.217 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2023-11-15T05:00:04.220 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2023-11-15T05:00:04.339 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2023-11-15T05:00:04.516 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:04.792 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:05.500 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:00:05.501 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:05.501 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2023-11-15T05:00:05.501 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:00:05.501 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:00:05.501 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2023-11-15T05:00:05.701 DEBUG:teuthology.parallel:result is None 2023-11-15T05:00:05.701 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:00:05.702 INFO:teuthology.packaging:ref: None 2023-11-15T05:00:05.702 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:00:05.702 INFO:teuthology.packaging:branch: None 2023-11-15T05:00:05.703 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:00:05.898 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:00:05.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:00:06.060 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T05:00:06.087 INFO:teuthology.orchestra.run.smithi086.stdout:16.2.4-0.el8 2023-11-15T05:00:06.088 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T05:00:06.088 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T05:00:06.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:00:06.089 INFO:teuthology.packaging:ref: None 2023-11-15T05:00:06.089 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:00:06.090 INFO:teuthology.packaging:branch: None 2023-11-15T05:00:06.090 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:00:06.290 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:00:06.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:00:06.517 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T05:00:06.544 INFO:teuthology.orchestra.run.smithi154.stdout:16.2.4-0.el8 2023-11-15T05:00:06.545 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T05:00:06.545 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T05:00:06.546 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T05:00:06.546 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:00:06.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T05:00:06.580 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:00:06.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T05:00:06.617 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T05:00:06.618 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:00:06.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T05:00:06.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T05:00:06.722 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:00:06.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T05:00:06.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T05:00:06.824 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T05:00:06.825 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:00:06.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T05:00:06.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T05:00:06.924 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:00:06.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T05:00:06.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T05:00:07.024 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T05:00:07.025 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:00:07.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T05:00:07.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T05:00:07.121 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:00:07.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T05:00:07.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T05:00:07.219 INFO:teuthology.run_tasks:Running task print... 2023-11-15T05:00:07.228 INFO:teuthology.task.print:**** done install task... 2023-11-15T05:00:07.229 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T05:00:07.313 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T05:00:07.314 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T05:00:07.315 INFO:tasks.cephadm:Cluster fsid is d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:07.315 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T05:00:07.315 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T05:00:07.316 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi086': '172.21.15.86', 'mon.smithi154': '172.21.15.154'} 2023-11-15T05:00:07.316 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T05:00:07.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2023-11-15T05:00:07.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2023-11-15T05:00:07.376 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T05:00:07.376 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.616 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 05:00 /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.616 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.758 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 05:00 /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.759 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.779 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T05:00:07.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2023-11-15T05:00:07.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2023-11-15T05:00:07.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2023-11-15T05:00:07.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2023-11-15T05:00:07.944 INFO:tasks.cephadm:Writing seed config... 2023-11-15T05:00:07.945 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T05:00:07.946 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T05:00:07.946 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T05:00:07.946 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T05:00:07.947 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T05:00:07.947 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T05:00:07.947 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T05:00:07.947 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T05:00:07.948 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T05:00:07.948 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T05:00:07.948 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T05:00:07.948 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T05:00:07.949 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T05:00:07.949 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T05:00:07.949 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T05:00:07.949 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T05:00:07.950 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T05:00:07.950 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T05:00:07.950 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T05:00:07.951 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T05:00:07.951 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T05:00:07.951 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T05:00:07.951 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T05:00:07.952 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T05:00:07.952 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T05:00:07.952 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T05:00:07.952 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T05:00:07.953 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T05:00:07.953 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T05:00:07.953 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T05:00:07.953 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T05:00:07.954 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T05:00:07.954 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T05:00:07.954 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T05:00:07.955 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T05:00:07.957 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T05:00:07.957 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T05:00:07.958 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T05:00:07.959 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:00:07.959 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T05:00:07.977 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d8947496-8373-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T05:00:07.978 DEBUG:teuthology.orchestra.run.smithi086:mon.smithi086> sudo journalctl -f -n 0 -u ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service 2023-11-15T05:00:08.020 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T05:00:08.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.86 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:00:08.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:-- Logs begin at Wed 2023-11-15 04:48:29 UTC. -- 2023-11-15T05:00:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:00:09.096 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: 4.6.1 2023-11-15T05:00:09.106 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying podman|docker is present... 2023-11-15T05:00:09.106 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:00:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: 4.6.1 2023-11-15T05:00:09.139 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying lvm2 is present... 2023-11-15T05:00:09.139 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying time synchronization is in place... 2023-11-15T05:00:09.139 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T05:00:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T05:00:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active chrony.service 2023-11-15T05:00:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:00:09.154 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T05:00:09.159 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: enabled 2023-11-15T05:00:09.159 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T05:00:09.165 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: active 2023-11-15T05:00:09.166 INFO:teuthology.orchestra.run.smithi086.stderr:Unit chronyd.service is enabled and running 2023-11-15T05:00:09.166 INFO:teuthology.orchestra.run.smithi086.stderr:Repeating the final host check... 2023-11-15T05:00:09.166 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:00:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: 4.6.1 2023-11-15T05:00:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:podman|docker (/bin/podman) is present 2023-11-15T05:00:09.195 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl is present 2023-11-15T05:00:09.196 INFO:teuthology.orchestra.run.smithi086.stderr:lvcreate is present 2023-11-15T05:00:09.196 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T05:00:09.200 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T05:00:09.201 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active chrony.service 2023-11-15T05:00:09.206 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:00:09.207 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T05:00:09.212 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: enabled 2023-11-15T05:00:09.213 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T05:00:09.218 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: active 2023-11-15T05:00:09.219 INFO:teuthology.orchestra.run.smithi086.stderr:Unit chronyd.service is enabled and running 2023-11-15T05:00:09.219 INFO:teuthology.orchestra.run.smithi086.stderr:Host looks OK 2023-11-15T05:00:09.219 INFO:teuthology.orchestra.run.smithi086.stderr:Cluster fsid: d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:09.219 INFO:teuthology.orchestra.run.smithi086.stderr:Acquiring lock 140399640463848 on /run/cephadm/d8947496-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:00:09.220 INFO:teuthology.orchestra.run.smithi086.stderr:Lock 140399640463848 acquired on /run/cephadm/d8947496-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:00:09.221 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying IP 172.21.15.86 port 3300 ... 2023-11-15T05:00:09.221 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying IP 172.21.15.86 port 6789 ... 2023-11-15T05:00:09.221 INFO:teuthology.orchestra.run.smithi086.stderr:Base mon IP is 172.21.15.86, final addrv is [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2023-11-15T05:00:09.221 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /sbin/ip route ls 2023-11-15T05:00:09.222 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.86 metric 100 2023-11-15T05:00:09.223 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.86 metric 100 2023-11-15T05:00:09.223 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T05:00:09.226 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T05:00:09.226 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-15T05:00:09.226 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2023-11-15T05:00:09.226 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T05:00:09.228 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T05:00:09.228 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T05:00:09.229 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T05:00:09.229 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-15T05:00:09.229 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2023-11-15T05:00:09.229 INFO:teuthology.orchestra.run.smithi086.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T05:00:09.230 INFO:teuthology.orchestra.run.smithi086.stderr:Mon IP 172.21.15.86 is in CIDR network 172.21.0.0/20 2023-11-15T05:00:09.230 INFO:teuthology.orchestra.run.smithi086.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T05:00:09.230 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T05:00:09.231 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T05:00:09.322 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T05:00:15.806 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Getting image source signatures 2023-11-15T05:00:15.813 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T05:00:15.813 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T05:00:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T05:00:27.468 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T05:00:27.713 INFO:teuthology.orchestra.run.smithi086.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T05:00:27.725 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T05:00:28.277 INFO:teuthology.orchestra.run.smithi086.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T05:00:28.621 INFO:teuthology.orchestra.run.smithi086.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T05:00:28.621 INFO:teuthology.orchestra.run.smithi086.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T05:00:28.621 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T05:00:28.914 INFO:teuthology.orchestra.run.smithi086.stderr:stat: 167 167 2023-11-15T05:00:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:Creating initial keys... 2023-11-15T05:00:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:00:29.509 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-authtool: AQBtUFRlLfsLHRAAAxHzX3fZ+o3DQGjQBlM/PA== 2023-11-15T05:00:29.839 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:00:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-authtool: AQBuUFRlyhz1BhAA/MDaAtDZjkvyrdDiJFl9yw== 2023-11-15T05:00:30.483 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:00:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-authtool: AQBuUFRlhrIAKxAA7NJgMEks/olmOBVo+w7hbA== 2023-11-15T05:00:31.067 INFO:teuthology.orchestra.run.smithi086.stderr:Creating initial monmap... 2023-11-15T05:00:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmph_6hv_09:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 --addv smithi086 [v2:172.21.15.86:3300,v1:172.21.15.86:6789] /tmp/monmap 2023-11-15T05:00:31.326 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T05:00:31.328 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:31.328 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T05:00:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:monmaptool for smithi086 [v2:172.21.15.86:3300,v1:172.21.15.86:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T05:00:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/monmaptool: set fsid to d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T05:00:31.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:00:31.680 INFO:teuthology.orchestra.run.smithi086.stderr:Creating mon... 2023-11-15T05:00:31.680 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/mon.smithi086:/var/lib/ceph/mon/ceph-smithi086:z -v /tmp/ceph-tmp7_lldgnb:/tmp/keyring:z -v /tmp/ceph-tmph_6hv_09:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi086 --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T05:00:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.944+0000 7f61a2965700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T05:00:31.996 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.944+0000 7f61a2965700 1 imported monmap: 2023-11-15T05:00:31.997 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T05:00:31.997 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: fsid d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:31.997 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T05:00:31.304747+0000 2023-11-15T05:00:31.997 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: created 2023-11-15T05:00:31.304747+0000 2023-11-15T05:00:31.998 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T05:00:31.998 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T05:00:31.998 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.smithi086 2023-11-15T05:00:31.998 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:31.998 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.944+0000 7f61a2965700 0 /usr/bin/ceph-mon: set fsid to d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Compile date May 13 2021 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: DB SUMMARY 2023-11-15T05:00:31.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 0, files: 2023-11-15T05:00:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 2023-11-15T05:00:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T05:00:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T05:00:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T05:00:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.env: 0x557f2f9801c0 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.fs: Posix File System 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.info_log: 0x557f311a6ec0 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.statistics: (nil) 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.use_fsync: 0 2023-11-15T05:00:32.002 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T05:00:32.003 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T05:00:32.003 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T05:00:32.003 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T05:00:32.003 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T05:00:32.003 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T05:00:32.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.db_log_dir: 2023-11-15T05:00:32.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi086/store.db 2023-11-15T05:00:32.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T05:00:32.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T05:00:32.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T05:00:32.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T05:00:32.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T05:00:32.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T05:00:32.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T05:00:32.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T05:00:32.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.write_buffer_manager: 0x557f311ad530 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T05:00:32.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T05:00:32.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T05:00:32.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T05:00:32.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T05:00:32.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T05:00:32.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T05:00:32.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.unordered_write: 0 2023-11-15T05:00:32.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T05:00:32.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T05:00:32.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T05:00:32.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T05:00:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.row_cache: None 2023-11-15T05:00:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.wal_filter: None 2023-11-15T05:00:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T05:00:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T05:00:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T05:00:32.012 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T05:00:32.012 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T05:00:32.013 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T05:00:32.013 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T05:00:32.013 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T05:00:32.013 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T05:00:32.013 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T05:00:32.014 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T05:00:32.014 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T05:00:32.014 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T05:00:32.014 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T05:00:32.015 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T05:00:32.015 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T05:00:32.015 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T05:00:32.015 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T05:00:32.015 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T05:00:32.016 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T05:00:32.016 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T05:00:32.016 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.max_open_files: -1 2023-11-15T05:00:32.016 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T05:00:32.017 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T05:00:32.017 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T05:00:32.017 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T05:00:32.017 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Compression algorithms supported: 2023-11-15T05:00:32.017 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T05:00:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kZSTD supported: 0 2023-11-15T05:00:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T05:00:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T05:00:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T05:00:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T05:00:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T05:00:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T05:00:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T05:00:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.945+0000 7f61a2965700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T05:00:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000001 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T05:00:32.020 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.merge_operator: 2023-11-15T05:00:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_filter: None 2023-11-15T05:00:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T05:00:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T05:00:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T05:00:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557f310be120) 2023-11-15T05:00:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T05:00:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T05:00:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T05:00:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T05:00:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T05:00:32.023 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T05:00:32.024 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_cache: 0x557f310f4d10 2023-11-15T05:00:32.024 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T05:00:32.024 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T05:00:32.024 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T05:00:32.024 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T05:00:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T05:00:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T05:00:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T05:00:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T05:00:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T05:00:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T05:00:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T05:00:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T05:00:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T05:00:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T05:00:32.027 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T05:00:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T05:00:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T05:00:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T05:00:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T05:00:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression: NoCompression 2023-11-15T05:00:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T05:00:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T05:00:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T05:00:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.num_levels: 7 2023-11-15T05:00:32.030 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T05:00:32.030 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T05:00:32.030 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T05:00:32.030 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T05:00:32.030 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T05:00:32.031 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T05:00:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T05:00:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T05:00:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:00:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T05:00:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T05:00:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T05:00:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T05:00:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T05:00:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T05:00:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T05:00:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T05:00:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T05:00:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T05:00:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T05:00:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T05:00:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T05:00:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T05:00:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T05:00:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T05:00:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T05:00:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T05:00:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T05:00:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T05:00:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T05:00:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T05:00:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T05:00:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T05:00:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T05:00:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T05:00:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T05:00:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T05:00:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T05:00:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T05:00:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.ttl: 2592000 2023-11-15T05:00:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T05:00:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T05:00:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.946+0000 7f61a2965700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T05:00:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.947+0000 7f61a2965700 4 rocksdb: DB pointer 0x557f311b9800 2023-11-15T05:00:32.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.947+0000 7f618b7fe700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T05:00:32.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.947+0000 7f618b7fe700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T05:00:32.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T05:00:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T05:00:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T05:00:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:00:32.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T05:00:32.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T05:00:32.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:00:32.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:00:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:00:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:00:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:00:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:00:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:00:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:00:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:00:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:00:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:00:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:00:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:00:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:00:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:00:32.055 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:32.055 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:32.055 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:00:32.055 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.056 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:00:32.056 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: 2023-11-15T05:00:32.056 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.948+0000 7f61a2965700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T05:00:32.056 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.948+0000 7f61a2965700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T05:00:32.056 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:00:31.948+0000 7f61a2965700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi086 for mon.smithi086 2023-11-15T05:00:32.299 INFO:teuthology.orchestra.run.smithi086.stderr:create mon.smithi086 on 2023-11-15T05:00:32.300 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:32.304 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl disable ceph.target 2023-11-15T05:00:32.311 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T05:00:32.397 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl enable ceph.target 2023-11-15T05:00:32.403 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T05:00:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl start ceph.target 2023-11-15T05:00:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl enable ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target 2023-11-15T05:00:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target. 2023-11-15T05:00:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target. 2023-11-15T05:00:32.575 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target 2023-11-15T05:00:32.582 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl daemon-reload 2023-11-15T05:00:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:00:32.695 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl reset-failed ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:00:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Failed to reset failed state of unit ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service: Unit ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service not loaded. 2023-11-15T05:00:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl enable ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:00:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target.wants/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service → /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T05:00:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:00:33.071 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:00:33.080 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: disabled 2023-11-15T05:00:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:00:33.088 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:00:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:firewalld.service is not enabled 2023-11-15T05:00:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T05:00:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T05:00:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mon to start... 2023-11-15T05:00:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mon... 2023-11-15T05:00:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/mon.smithi086:/var/lib/ceph/mon/ceph-smithi086:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T05:00:33.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.461+0000 7fe0d1bd4700 1 Processor -- start 2023-11-15T05:00:33.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.461+0000 7fe0d1bd4700 1 -- start start 2023-11-15T05:00:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.461+0000 7fe0d1bd4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.461+0000 7fe0d1bd4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0fb490 con 0x7fe0cc0fab80 2023-11-15T05:00:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.462+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.462+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51888/0 (socket says 172.21.15.86:51888) 2023-11-15T05:00:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.462+0000 7fe0cb7fe700 1 -- 172.21.15.86:0/3406830976 learned_addr learned my addr 172.21.15.86:0/3406830976 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:33.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.462+0000 7fe0cb7fe700 1 -- 172.21.15.86:0/3406830976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0fb5d0 con 0x7fe0cc0fab80 2023-11-15T05:00:33.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.462+0000 7fe0cb7fe700 1 --2- 172.21.15.86:0/3406830976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe0c000b5f0 tx=0x7fe0c0006ee0).ready entity=mon.0 client_cookie=18108e9c6dc3a913 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:33.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0ca7fc700 1 -- 172.21.15.86:0/3406830976 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c000d600 con 0x7fe0cc0fab80 2023-11-15T05:00:33.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0ca7fc700 1 -- 172.21.15.86:0/3406830976 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c000d760 con 0x7fe0cc0fab80 2023-11-15T05:00:33.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0ca7fc700 1 -- 172.21.15.86:0/3406830976 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c000da30 con 0x7fe0cc0fab80 2023-11-15T05:00:33.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/3406830976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 msgr2=0x7fe0cc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:33.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 --2- 172.21.15.86:0/3406830976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe0c000b5f0 tx=0x7fe0c0006ee0).stop 2023-11-15T05:00:33.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/3406830976 shutdown_connections 2023-11-15T05:00:33.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 --2- 172.21.15.86:0/3406830976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc0fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/3406830976 >> 172.21.15.86:0/3406830976 conn(0x7fe0cc0f63c0 msgr2=0x7fe0cc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/3406830976 shutdown_connections 2023-11-15T05:00:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.463+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/3406830976 wait complete. 2023-11-15T05:00:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0d1bd4700 1 Processor -- start 2023-11-15T05:00:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0d1bd4700 1 -- start start 2023-11-15T05:00:33.471 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0d1bd4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:33.471 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0d1bd4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc109640 con 0x7fe0cc0fab80 2023-11-15T05:00:33.471 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:33.471 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0cb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51902/0 (socket says 172.21.15.86:51902) 2023-11-15T05:00:33.472 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0cb7fe700 1 -- 172.21.15.86:0/466510010 learned_addr learned my addr 172.21.15.86:0/466510010 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:33.472 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.464+0000 7fe0cb7fe700 1 -- 172.21.15.86:0/466510010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c000b200 con 0x7fe0cc0fab80 2023-11-15T05:00:33.473 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0cb7fe700 1 --2- 172.21.15.86:0/466510010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe0c0006c70 tx=0x7fe0c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:33.473 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c0011b40 con 0x7fe0cc0fab80 2023-11-15T05:00:33.473 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0011ca0 con 0x7fe0cc0fab80 2023-11-15T05:00:33.473 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc109840 con 0x7fe0cc0fab80 2023-11-15T05:00:33.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0c001d430 con 0x7fe0cc0fab80 2023-11-15T05:00:33.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc109c50 con 0x7fe0cc0fab80 2023-11-15T05:00:33.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe0c000f070 con 0x7fe0cc0fab80 2023-11-15T05:00:33.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.465+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe0c0027460 con 0x7fe0cc0fab80 2023-11-15T05:00:33.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.466+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc045ad0 con 0x7fe0cc0fab80 2023-11-15T05:00:33.475 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.468+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe0c0016070 con 0x7fe0cc0fab80 2023-11-15T05:00:33.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.087016+0000 mon.smithi086 (mon.0) 1 : cluster [INF] mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2023-11-15T05:00:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.535+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe0cc102b50 con 0x7fe0cc0fab80 2023-11-15T05:00:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.535+0000 7fe0b3fff700 1 -- 172.21.15.86:0/466510010 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe0c001d640 con 0x7fe0cc0fab80 2023-11-15T05:00:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: cluster: 2023-11-15T05:00:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: id: d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T05:00:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: services: 2023-11-15T05:00:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi086 (age 0.429285s) 2023-11-15T05:00:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T05:00:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T05:00:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: data: 2023-11-15T05:00:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T05:00:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T05:00:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T05:00:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: pgs: 2023-11-15T05:00:33.542 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 msgr2=0x7fe0cc109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 --2- 172.21.15.86:0/466510010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe0c0006c70 tx=0x7fe0c0016000).stop 2023-11-15T05:00:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 shutdown_connections 2023-11-15T05:00:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 --2- 172.21.15.86:0/466510010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0cc0fab80 0x7fe0cc109150 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 >> 172.21.15.86:0/466510010 conn(0x7fe0cc0f63c0 msgr2=0x7fe0cc0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:33.544 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 shutdown_connections 2023-11-15T05:00:33.544 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:33.536+0000 7fe0d1bd4700 1 -- 172.21.15.86:0/466510010 wait complete. 2023-11-15T05:00:33.869 INFO:teuthology.orchestra.run.smithi086.stderr:mon is available 2023-11-15T05:00:33.870 INFO:teuthology.orchestra.run.smithi086.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T05:00:33.870 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z -v /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/mon.smithi086:/var/lib/ceph/mon/ceph-smithi086:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi086/config 2023-11-15T05:00:34.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.284+0000 7f2047a63700 1 Processor -- start 2023-11-15T05:00:34.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f2047a63700 1 -- start start 2023-11-15T05:00:34.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f2047a63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f2047a63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20400fb490 con 0x7f20400fab80 2023-11-15T05:00:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f20457ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f20457ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51908/0 (socket says 172.21.15.86:51908) 2023-11-15T05:00:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.285+0000 7f20457ff700 1 -- 172.21.15.86:0/838788918 learned_addr learned my addr 172.21.15.86:0/838788918 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f20457ff700 1 -- 172.21.15.86:0/838788918 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20400fb5d0 con 0x7f20400fab80 2023-11-15T05:00:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f20457ff700 1 --2- 172.21.15.86:0/838788918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f203400b5f0 tx=0x7f2034006ee0).ready entity=mon.0 client_cookie=a5621428feaf0fa server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2033fff700 1 -- 172.21.15.86:0/838788918 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203400d600 con 0x7f20400fab80 2023-11-15T05:00:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2033fff700 1 -- 172.21.15.86:0/838788918 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203400d760 con 0x7f20400fab80 2023-11-15T05:00:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2033fff700 1 -- 172.21.15.86:0/838788918 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203400da30 con 0x7f20400fab80 2023-11-15T05:00:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2047a63700 1 -- 172.21.15.86:0/838788918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 msgr2=0x7f20400fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:34.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2047a63700 1 --2- 172.21.15.86:0/838788918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f203400b5f0 tx=0x7f2034006ee0).stop 2023-11-15T05:00:34.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2047a63700 1 -- 172.21.15.86:0/838788918 shutdown_connections 2023-11-15T05:00:34.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2047a63700 1 --2- 172.21.15.86:0/838788918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f20400fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:34.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.286+0000 7f2047a63700 1 -- 172.21.15.86:0/838788918 >> 172.21.15.86:0/838788918 conn(0x7f20400f63c0 msgr2=0x7f20400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:34.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 -- 172.21.15.86:0/838788918 shutdown_connections 2023-11-15T05:00:34.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 -- 172.21.15.86:0/838788918 wait complete. 2023-11-15T05:00:34.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 Processor -- start 2023-11-15T05:00:34.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 -- start start 2023-11-15T05:00:34.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:34.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f2047a63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204010a710 con 0x7f20400fab80 2023-11-15T05:00:34.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.287+0000 7f20457ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:34.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f20457ff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51922/0 (socket says 172.21.15.86:51922) 2023-11-15T05:00:34.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f20457ff700 1 -- 172.21.15.86:0/4044587860 learned_addr learned my addr 172.21.15.86:0/4044587860 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:34.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f20457ff700 1 -- 172.21.15.86:0/4044587860 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203400b200 con 0x7f20400fab80 2023-11-15T05:00:34.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f20457ff700 1 --2- 172.21.15.86:0/4044587860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f203400c010 tx=0x7f2034016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2034011cc0 con 0x7f20400fab80 2023-11-15T05:00:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2034011e20 con 0x7f20400fab80 2023-11-15T05:00:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f203401d590 con 0x7f20400fab80 2023-11-15T05:00:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204010a910 con 0x7f20400fab80 2023-11-15T05:00:34.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.288+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204010ad20 con 0x7f20400fab80 2023-11-15T05:00:34.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.289+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f203400f070 con 0x7f20400fab80 2023-11-15T05:00:34.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.289+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2034027460 con 0x7f20400fab80 2023-11-15T05:00:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.289+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2040045ad0 con 0x7f20400fab80 2023-11-15T05:00:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.292+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2034016070 con 0x7f20400fab80 2023-11-15T05:00:34.356 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.355+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f20400fbfb0 con 0x7f20400fab80 2023-11-15T05:00:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.358+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f203401d800 con 0x7f20400fab80 2023-11-15T05:00:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.358+0000 7f2031ffb700 1 -- 172.21.15.86:0/4044587860 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f2034036020 con 0x7f20400fab80 2023-11-15T05:00:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: [global] 2023-11-15T05:00:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: fsid = d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2023-11-15T05:00:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T05:00:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T05:00:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T05:00:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T05:00:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: [mgr] 2023-11-15T05:00:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T05:00:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: [osd] 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T05:00:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T05:00:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T05:00:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 msgr2=0x7f204010a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 --2- 172.21.15.86:0/4044587860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f203400c010 tx=0x7f2034016000).stop 2023-11-15T05:00:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 shutdown_connections 2023-11-15T05:00:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 --2- 172.21.15.86:0/4044587860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20400fab80 0x7f204010a220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 >> 172.21.15.86:0/4044587860 conn(0x7f20400f63c0 msgr2=0x7f20400fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 shutdown_connections 2023-11-15T05:00:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:34.359+0000 7f2047a63700 1 -- 172.21.15.86:0/4044587860 wait complete. 2023-11-15T05:00:34.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.107795+0000 mon.smithi086 (mon.0) 2 : cluster [INF] mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2023-11-15T05:00:34.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.108053+0000 mon.smithi086 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} 2023-11-15T05:00:34.381 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.114299+0000 mon.smithi086 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T05:00:34.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.122373+0000 mon.smithi086 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T05:00:34.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: cluster 2023-11-15T05:00:33.123419+0000 mon.smithi086 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T05:00:34.382 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: audit 2023-11-15T05:00:33.537046+0000 mon.smithi086 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.86:0/466510010' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T05:00:34.711 INFO:teuthology.orchestra.run.smithi086.stderr:Generating new minimal ceph.conf... 2023-11-15T05:00:34.711 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z -v /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/mon.smithi086:/var/lib/ceph/mon/ceph-smithi086:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi086/config 2023-11-15T05:00:35.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.042+0000 7f79117a4700 1 Processor -- start 2023-11-15T05:00:35.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.042+0000 7f79117a4700 1 -- start start 2023-11-15T05:00:35.045 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.042+0000 7f79117a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.042+0000 7f79117a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0fedd0 con 0x7f790c070430 2023-11-15T05:00:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f790affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f790affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51930/0 (socket says 172.21.15.86:51930) 2023-11-15T05:00:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f790affd700 1 -- 172.21.15.86:0/2953209613 learned_addr learned my addr 172.21.15.86:0/2953209613 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f790affd700 1 -- 172.21.15.86:0/2953209613 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0fef10 con 0x7f790c070430 2023-11-15T05:00:35.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f790affd700 1 --2- 172.21.15.86:0/2953209613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f790000b5f0 tx=0x7f7900006ee0).ready entity=mon.0 client_cookie=2d73557a2814d52f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:35.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f7909ffb700 1 -- 172.21.15.86:0/2953209613 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f790000d600 con 0x7f790c070430 2023-11-15T05:00:35.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f7909ffb700 1 -- 172.21.15.86:0/2953209613 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f790000dbd0 con 0x7f790c070430 2023-11-15T05:00:35.047 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.043+0000 7f7909ffb700 1 -- 172.21.15.86:0/2953209613 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7900011c00 con 0x7f790c070430 2023-11-15T05:00:35.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 -- 172.21.15.86:0/2953209613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 msgr2=0x7f790c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:35.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 --2- 172.21.15.86:0/2953209613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f790000b5f0 tx=0x7f7900006ee0).stop 2023-11-15T05:00:35.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 -- 172.21.15.86:0/2953209613 shutdown_connections 2023-11-15T05:00:35.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 --2- 172.21.15.86:0/2953209613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:35.049 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 -- 172.21.15.86:0/2953209613 >> 172.21.15.86:0/2953209613 conn(0x7f790c0f63c0 msgr2=0x7f790c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:35.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 -- 172.21.15.86:0/2953209613 shutdown_connections 2023-11-15T05:00:35.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.044+0000 7f79117a4700 1 -- 172.21.15.86:0/2953209613 wait complete. 2023-11-15T05:00:35.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 Processor -- start 2023-11-15T05:00:35.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 -- start start 2023-11-15T05:00:35.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:35.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0fa7e0 con 0x7f790c070430 2023-11-15T05:00:35.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f790affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:35.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f790affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51944/0 (socket says 172.21.15.86:51944) 2023-11-15T05:00:35.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f790affd700 1 -- 172.21.15.86:0/3927051476 learned_addr learned my addr 172.21.15.86:0/3927051476 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:35.051 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f790affd700 1 -- 172.21.15.86:0/3927051476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790000b200 con 0x7f790c070430 2023-11-15T05:00:35.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f790affd700 1 --2- 172.21.15.86:0/3927051476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f790000e040 tx=0x7f7900006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:35.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f790000d9f0 con 0x7f790c070430 2023-11-15T05:00:35.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c0fa980 con 0x7f790c070430 2023-11-15T05:00:35.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c0fad90 con 0x7f790c070430 2023-11-15T05:00:35.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f790001a9f0 con 0x7f790c070430 2023-11-15T05:00:35.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.045+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f790001b8c0 con 0x7f790c070430 2023-11-15T05:00:35.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.046+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f790000f070 con 0x7f790c070430 2023-11-15T05:00:35.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.046+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7900025b30 con 0x7f790c070430 2023-11-15T05:00:35.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.047+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c045ad0 con 0x7f790c070430 2023-11-15T05:00:35.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.049+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7900016070 con 0x7f790c070430 2023-11-15T05:00:35.115 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.113+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f790c104820 con 0x7f790c070430 2023-11-15T05:00:35.115 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.114+0000 7f78f37fe700 1 -- 172.21.15.86:0/3927051476 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f790001bdc0 con 0x7f790c070430 2023-11-15T05:00:35.116 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.114+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 msgr2=0x7f790c0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:35.116 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.114+0000 7f79117a4700 1 --2- 172.21.15.86:0/3927051476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f790000e040 tx=0x7f7900006ee0).stop 2023-11-15T05:00:35.117 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.114+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 shutdown_connections 2023-11-15T05:00:35.117 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.115+0000 7f79117a4700 1 --2- 172.21.15.86:0/3927051476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f790c070430 0x7f790c0fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:35.117 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.115+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 >> 172.21.15.86:0/3927051476 conn(0x7f790c0f63c0 msgr2=0x7f790c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:35.117 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.115+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 shutdown_connections 2023-11-15T05:00:35.117 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:35.115+0000 7f79117a4700 1 -- 172.21.15.86:0/3927051476 wait complete. 2023-11-15T05:00:35.387 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: audit 2023-11-15T05:00:34.356749+0000 mon.smithi086 (mon.0) 8 : audit [INF] from='client.? 172.21.15.86:0/4044587860' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T05:00:35.388 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: audit 2023-11-15T05:00:34.359791+0000 mon.smithi086 (mon.0) 9 : audit [INF] from='client.? 172.21.15.86:0/4044587860' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T05:00:35.388 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: audit 2023-11-15T05:00:35.115483+0000 mon.smithi086 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.86:0/3927051476' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T05:00:35.452 INFO:teuthology.orchestra.run.smithi086.stderr:Restarting the monitor... 2023-11-15T05:00:35.453 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl restart ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:00:35.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for d8947496-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T05:00:35.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: debug 2023-11-15T05:00:35.524+0000 7f69fa665700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T05:00:35.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[111911]: debug 2023-11-15T05:00:35.524+0000 7f69fa665700 -1 mon.smithi086@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T05:00:36.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 podman[112234]: ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086 2023-11-15T05:00:36.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 systemd[1]: ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service: Succeeded. 2023-11-15T05:00:36.188 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 systemd[1]: Stopped Ceph mon.smithi086 for d8947496-8373-11ee-8dbe-212e2dc638e7. 2023-11-15T05:00:36.188 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:35 smithi086 systemd[1]: Starting Ceph mon.smithi086 for d8947496-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T05:00:36.252 INFO:teuthology.orchestra.run.smithi086.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T05:00:36.253 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T05:00:36.620 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.615+0000 7f47a0be6700 1 Processor -- start 2023-11-15T05:00:36.621 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.615+0000 7f47a0be6700 1 -- start start 2023-11-15T05:00:36.621 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f47a0be6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:36.621 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f47a0be6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c0fb490 con 0x7f479c0fab80 2023-11-15T05:00:36.621 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f479a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:36.622 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f479a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51948/0 (socket says 172.21.15.86:51948) 2023-11-15T05:00:36.622 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f479a59c700 1 -- 172.21.15.86:0/3654940331 learned_addr learned my addr 172.21.15.86:0/3654940331 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:36.622 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f479a59c700 1 -- 172.21.15.86:0/3654940331 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479c0fb5d0 con 0x7f479c0fab80 2023-11-15T05:00:36.623 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.616+0000 7f479a59c700 1 --2- 172.21.15.86:0/3654940331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f479000b5f0 tx=0x7f4790006ee0).ready entity=mon.0 client_cookie=aa365f7e0697d035 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:36.623 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f479959a700 1 -- 172.21.15.86:0/3654940331 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479000d600 con 0x7f479c0fab80 2023-11-15T05:00:36.623 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f479959a700 1 -- 172.21.15.86:0/3654940331 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f479000dbd0 con 0x7f479c0fab80 2023-11-15T05:00:36.623 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f479959a700 1 -- 172.21.15.86:0/3654940331 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4790011c00 con 0x7f479c0fab80 2023-11-15T05:00:36.623 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 -- 172.21.15.86:0/3654940331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 msgr2=0x7f479c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:36.625 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 --2- 172.21.15.86:0/3654940331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f479000b5f0 tx=0x7f4790006ee0).stop 2023-11-15T05:00:36.625 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 -- 172.21.15.86:0/3654940331 shutdown_connections 2023-11-15T05:00:36.625 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 --2- 172.21.15.86:0/3654940331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c0fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:36.626 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 -- 172.21.15.86:0/3654940331 >> 172.21.15.86:0/3654940331 conn(0x7f479c0f63c0 msgr2=0x7f479c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:36.626 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 -- 172.21.15.86:0/3654940331 shutdown_connections 2023-11-15T05:00:36.626 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.617+0000 7f47a0be6700 1 -- 172.21.15.86:0/3654940331 wait complete. 2023-11-15T05:00:36.626 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f47a0be6700 1 Processor -- start 2023-11-15T05:00:36.626 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f47a0be6700 1 -- start start 2023-11-15T05:00:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f47a0be6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f47a0be6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479c10aee0 con 0x7f479c0fab80 2023-11-15T05:00:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f479a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f479a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51960/0 (socket says 172.21.15.86:51960) 2023-11-15T05:00:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f479a59c700 1 -- 172.21.15.86:0/733607313 learned_addr learned my addr 172.21.15.86:0/733607313 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:36.628 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f479a59c700 1 -- 172.21.15.86:0/733607313 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479000b200 con 0x7f479c0fab80 2023-11-15T05:00:36.628 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.618+0000 7f479a59c700 1 --2- 172.21.15.86:0/733607313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f479000b1d0 tx=0x7f4790006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:36.628 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479001b410 con 0x7f479c0fab80 2023-11-15T05:00:36.628 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f479001b9e0 con 0x7f479c0fab80 2023-11-15T05:00:36.628 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479c10b0e0 con 0x7f479c0fab80 2023-11-15T05:00:36.629 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f479001a840 con 0x7f479c0fab80 2023-11-15T05:00:36.629 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479c10b4f0 con 0x7f479c0fab80 2023-11-15T05:00:36.629 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f479000f070 con 0x7f479c0fab80 2023-11-15T05:00:36.629 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.619+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4790025b80 con 0x7f479c0fab80 2023-11-15T05:00:36.629 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.620+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479c045ad0 con 0x7f479c0fab80 2023-11-15T05:00:36.630 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.622+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4790016070 con 0x7f479c0fab80 2023-11-15T05:00:36.686 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.684+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f479c104a40 con 0x7f479c0fab80 2023-11-15T05:00:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.687+0000 7f478affd700 1 -- 172.21.15.86:0/733607313 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4790025dc0 con 0x7f479c0fab80 2023-11-15T05:00:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 msgr2=0x7f479c10a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 --2- 172.21.15.86:0/733607313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f479000b1d0 tx=0x7f4790006ee0).stop 2023-11-15T05:00:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 shutdown_connections 2023-11-15T05:00:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 --2- 172.21.15.86:0/733607313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f479c0fab80 0x7f479c10a9f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 >> 172.21.15.86:0/733607313 conn(0x7f479c0f63c0 msgr2=0x7f479c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 shutdown_connections 2023-11-15T05:00:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:36.688+0000 7f47a0be6700 1 -- 172.21.15.86:0/733607313 wait complete. 2023-11-15T05:00:36.711 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.243+0000 7f88b5840700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T05:00:36.712 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.243+0000 7f88b5840700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T05:00:36.712 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.243+0000 7f88b5840700 0 pidfile_write: ignore empty --pid-file 2023-11-15T05:00:36.712 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.247+0000 7f88b5840700 0 load: jerasure load: lrc load: isa 2023-11-15T05:00:36.712 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T05:00:36.712 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.713 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T05:00:36.713 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Compile date May 13 2021 2023-11-15T05:00:36.713 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: DB SUMMARY 2023-11-15T05:00:36.713 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.713 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T05:00:36.714 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.714 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T05:00:36.714 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.714 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T05:00:36.714 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.715 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T05:00:36.715 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.715 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 000006.log size: 93434 ; 2023-11-15T05:00:36.715 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 bash[112306]: b2e9dbbe419122c615b25fbdc1d34c39e503ea8c27c14d1ff07c11d71ab687d9 2023-11-15T05:00:36.715 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.env: 0x5638573421c0 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.fs: Posix File System 2023-11-15T05:00:36.716 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.info_log: 0x563857dff960 2023-11-15T05:00:36.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T05:00:36.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.statistics: (nil) 2023-11-15T05:00:36.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.use_fsync: 0 2023-11-15T05:00:36.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T05:00:36.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T05:00:36.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T05:00:36.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T05:00:36.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T05:00:36.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T05:00:36.719 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T05:00:36.719 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T05:00:36.719 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T05:00:36.719 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T05:00:36.719 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T05:00:36.720 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.db_log_dir: 2023-11-15T05:00:36.720 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi086/store.db 2023-11-15T05:00:36.720 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T05:00:36.720 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T05:00:36.720 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T05:00:36.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T05:00:36.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T05:00:36.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T05:00:36.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T05:00:36.721 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T05:00:36.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T05:00:36.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T05:00:36.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.write_buffer_manager: 0x563858a74030 2023-11-15T05:00:36.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T05:00:36.722 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T05:00:36.723 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T05:00:36.723 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T05:00:36.723 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T05:00:36.723 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T05:00:36.723 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T05:00:36.724 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T05:00:36.724 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T05:00:36.724 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.unordered_write: 0 2023-11-15T05:00:36.726 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T05:00:36.726 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T05:00:36.727 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T05:00:36.727 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T05:00:36.727 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.row_cache: None 2023-11-15T05:00:36.727 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.wal_filter: None 2023-11-15T05:00:36.727 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T05:00:36.728 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T05:00:36.728 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T05:00:36.728 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T05:00:36.728 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T05:00:36.728 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T05:00:36.729 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T05:00:36.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T05:00:36.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T05:00:36.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T05:00:36.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T05:00:36.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T05:00:36.731 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T05:00:36.731 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T05:00:36.731 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T05:00:36.731 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T05:00:36.731 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_open_files: -1 2023-11-15T05:00:36.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T05:00:36.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T05:00:36.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T05:00:36.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T05:00:36.732 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Compression algorithms supported: 2023-11-15T05:00:36.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T05:00:36.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kZSTD supported: 0 2023-11-15T05:00:36.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T05:00:36.733 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T05:00:36.734 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T05:00:36.734 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T05:00:36.734 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T05:00:36.734 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T05:00:36.734 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T05:00:36.735 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000005 2023-11-15T05:00:36.735 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.735 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T05:00:36.735 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.735 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T05:00:36.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.merge_operator: 2023-11-15T05:00:36.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_filter: None 2023-11-15T05:00:36.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T05:00:36.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T05:00:36.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T05:00:36.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563857d16128) 2023-11-15T05:00:36.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cache_index_and_filter_blocks: 1 2023-11-15T05:00:36.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T05:00:36.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T05:00:36.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: pin_top_level_index_and_filter: 1 2023-11-15T05:00:36.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: index_type: 0 2023-11-15T05:00:36.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: data_block_index_type: 0 2023-11-15T05:00:36.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: index_shortening: 1 2023-11-15T05:00:36.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T05:00:36.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: hash_index_allow_collision: 1 2023-11-15T05:00:36.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: checksum: 1 2023-11-15T05:00:36.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: no_block_cache: 0 2023-11-15T05:00:36.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_cache: 0x563857d4cf10 2023-11-15T05:00:36.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_cache_name: BinnedLRUCache 2023-11-15T05:00:36.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_cache_options: 2023-11-15T05:00:36.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: capacity : 536870912 2023-11-15T05:00:36.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: num_shard_bits : 4 2023-11-15T05:00:36.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: strict_capacity_limit : 0 2023-11-15T05:00:36.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: high_pri_pool_ratio: 0.000 2023-11-15T05:00:36.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_cache_compressed: (nil) 2023-11-15T05:00:36.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: persistent_cache: (nil) 2023-11-15T05:00:36.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_size: 4096 2023-11-15T05:00:36.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_size_deviation: 10 2023-11-15T05:00:36.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_restart_interval: 16 2023-11-15T05:00:36.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: index_block_restart_interval: 1 2023-11-15T05:00:36.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: metadata_block_size: 4096 2023-11-15T05:00:36.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: partition_filters: 0 2023-11-15T05:00:36.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: use_delta_encoding: 1 2023-11-15T05:00:36.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T05:00:36.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: whole_key_filtering: 1 2023-11-15T05:00:36.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: verify_compression: 0 2023-11-15T05:00:36.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: read_amp_bytes_per_bit: 0 2023-11-15T05:00:36.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: format_version: 2 2023-11-15T05:00:36.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: enable_index_compression: 1 2023-11-15T05:00:36.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: block_align: 0 2023-11-15T05:00:36.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T05:00:36.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T05:00:36.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression: NoCompression 2023-11-15T05:00:36.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T05:00:36.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T05:00:36.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T05:00:36.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.num_levels: 7 2023-11-15T05:00:36.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T05:00:36.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T05:00:36.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T05:00:36.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T05:00:36.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T05:00:36.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T05:00:36.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T05:00:36.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:00:36.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T05:00:36.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T05:00:36.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T05:00:36.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T05:00:36.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T05:00:36.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:00:36.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T05:00:36.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T05:00:36.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T05:00:36.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T05:00:36.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T05:00:36.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T05:00:36.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T05:00:36.751 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T05:00:36.753 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T05:00:36.753 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T05:00:36.754 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T05:00:36.754 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T05:00:36.754 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T05:00:36.755 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T05:00:36.755 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T05:00:36.755 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T05:00:36.755 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T05:00:36.755 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T05:00:36.756 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T05:00:36.756 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T05:00:36.756 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T05:00:36.756 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T05:00:36.756 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T05:00:36.757 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T05:00:36.757 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T05:00:36.757 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T05:00:36.757 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T05:00:36.758 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T05:00:36.758 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T05:00:36.758 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T05:00:36.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T05:00:36.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T05:00:36.759 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T05:00:36.760 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T05:00:36.760 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T05:00:36.760 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T05:00:36.761 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T05:00:36.761 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T05:00:36.761 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T05:00:36.761 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T05:00:36.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.ttl: 2592000 2023-11-15T05:00:36.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.248+0000 7f88b5840700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T05:00:36.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.249+0000 7f88b5840700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T05:00:36.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.762 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.249+0000 7f88b5840700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T05:00:36.763 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.763 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.249+0000 7f88b5840700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024436250540, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T05:00:36.763 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.249+0000 7f88b5840700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T05:00:36.763 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.249+0000 7f88b5840700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.250+0000 7f88b5840700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024436251397, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 89214, "table_properties": {"data_size": 87329, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 80470, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700024436, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.250+0000 7f88b5840700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.250+0000 7f88b5840700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700024436251780, "job": 1, "event": "recovery_finished"} 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 systemd[1]: Started Ceph mon.smithi086 for d8947496-8373-11ee-8dbe-212e2dc638e7. 2023-11-15T05:00:36.764 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.251+0000 7f88b5840700 4 rocksdb: DB pointer 0x563857e11800 2023-11-15T05:00:36.765 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.251+0000 7f889c658700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T05:00:36.765 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.251+0000 7f889c658700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T05:00:36.765 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** DB Stats ** 2023-11-15T05:00:36.766 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:36.766 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T05:00:36.766 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T05:00:36.766 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:00:36.766 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T05:00:36.767 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T05:00:36.767 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:00:36.767 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.767 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** Compaction Stats [default] ** 2023-11-15T05:00:36.767 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:36.768 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:36.768 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.768 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.768 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.769 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.769 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** Compaction Stats [default] ** 2023-11-15T05:00:36.769 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:36.769 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:36.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:36.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:36.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:36.771 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:00:36.771 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:00:36.771 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:00:36.771 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Cumulative compaction: 0.00 GB write, 29.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:36.771 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Interval compaction: 0.00 GB write, 29.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:36.772 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:00:36.772 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.772 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:00:36.772 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.772 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** Compaction Stats [default] ** 2023-11-15T05:00:36.773 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:36.773 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:36.773 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.773 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.773 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:00:36.774 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.774 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** Compaction Stats [default] ** 2023-11-15T05:00:36.774 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:00:36.774 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:00:36.774 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.3 0.00 0.00 1 0.001 0 0 2023-11-15T05:00:36.775 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:00:36.775 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:36.775 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:00:36.775 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:00:36.775 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:00:36.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:00:36.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Cumulative compaction: 0.00 GB write, 29.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:36.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:00:36.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:00:36.776 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:00:36.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.251+0000 7f88b5840700 0 starting mon.smithi086 rank 0 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi086 fsid d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:36.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.252+0000 7f88b5840700 1 mon.smithi086@-1(???) e1 preinit fsid d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:00:36.777 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.252+0000 7f88b5840700 0 mon.smithi086@-1(???).mds e1 new map 2023-11-15T05:00:36.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.252+0000 7f88b5840700 0 mon.smithi086@-1(???).mds e1 print_map 2023-11-15T05:00:36.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: e1 2023-11-15T05:00:36.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T05:00:36.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T05:00:36.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: legacy client fscid: -1 2023-11-15T05:00:36.779 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.779 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: No filesystems configured 2023-11-15T05:00:36.779 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: 2023-11-15T05:00:36.779 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.253+0000 7f88b5840700 0 mon.smithi086@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T05:00:36.780 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.253+0000 7f88b5840700 0 mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:00:36.780 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.253+0000 7f88b5840700 0 mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:00:36.780 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.253+0000 7f88b5840700 0 mon.smithi086@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:00:36.780 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:34.356749+0000 mon.smithi086 (mon.0) 8 : audit [INF] from='client.? 172.21.15.86:0/4044587860' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T05:00:36.780 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:34.359791+0000 mon.smithi086 (mon.0) 9 : audit [INF] from='client.? 172.21.15.86:0/4044587860' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T05:00:36.781 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:35.115483+0000 mon.smithi086 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.86:0/3927051476' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T05:00:36.781 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.253+0000 7f88b5840700 1 mon.smithi086@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T05:00:36.781 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expand map: {default=false} 2023-11-15T05:00:36.781 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta from 'false' to 'false' 2023-11-15T05:00:36.781 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expanded map: {default=false} 2023-11-15T05:00:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expand map: {default=info} 2023-11-15T05:00:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta from 'info' to 'info' 2023-11-15T05:00:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expanded map: {default=info} 2023-11-15T05:00:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expand map: {default=daemon} 2023-11-15T05:00:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-15T05:00:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-15T05:00:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expand map: {default=debug} 2023-11-15T05:00:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-15T05:00:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:00:36.254+0000 7f88b5840700 20 expand_channel_meta expanded map: {default=debug} 2023-11-15T05:00:36.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:36.270179+0000 mon.smithi086 (mon.0) 1 : cluster [INF] mon.smithi086 is new leader, mons smithi086 in quorum (ranks 0) 2023-11-15T05:00:36.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:36.270271+0000 mon.smithi086 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} 2023-11-15T05:00:36.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:36.270325+0000 mon.smithi086 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T05:00:36.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:36.270378+0000 mon.smithi086 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T05:00:36.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:36.270927+0000 mon.smithi086 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T05:00:37.069 INFO:teuthology.orchestra.run.smithi086.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T05:00:37.070 INFO:teuthology.orchestra.run.smithi086.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:00:37.070 INFO:teuthology.orchestra.run.smithi086.stderr:Creating mgr... 2023-11-15T05:00:37.071 INFO:teuthology.orchestra.run.smithi086.stderr:Verifying port 9283 ... 2023-11-15T05:00:37.072 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl daemon-reload 2023-11-15T05:00:37.167 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo 2023-11-15T05:00:37.189 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl reset-failed ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo 2023-11-15T05:00:37.195 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Failed to reset failed state of unit ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo.service: Unit ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo.service not loaded. 2023-11-15T05:00:37.195 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl enable ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo 2023-11-15T05:00:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7.target.wants/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo.service → /etc/systemd/system/ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T05:00:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mgr.smithi086.hkfamo 2023-11-15T05:00:37.604 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:00:37.612 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: disabled 2023-11-15T05:00:37.613 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:00:37.620 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:00:37.621 INFO:teuthology.orchestra.run.smithi086.stderr:firewalld.service is not enabled 2023-11-15T05:00:37.621 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T05:00:37.621 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T05:00:37.621 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:00:37.628 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: disabled 2023-11-15T05:00:37.629 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:00:37.636 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:00:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:firewalld.service is not enabled 2023-11-15T05:00:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T05:00:37.637 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mgr to start... 2023-11-15T05:00:37.638 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mgr... 2023-11-15T05:00:37.638 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:00:37.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:36.689175+0000 mon.smithi086 (mon.0) 6 : audit [INF] from='client.? 172.21.15.86:0/733607313' entity='client.admin' 2023-11-15T05:00:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.997+0000 7fabb1f3e700 1 Processor -- start 2023-11-15T05:00:38.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.997+0000 7fabb1f3e700 1 -- start start 2023-11-15T05:00:38.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.997+0000 7fabb1f3e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:38.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.997+0000 7fabb1f3e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac070770 con 0x7fabac070350 2023-11-15T05:00:38.004 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51996/0 (socket says 172.21.15.86:51996) 2023-11-15T05:00:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabab7fe700 1 -- 172.21.15.86:0/2102309607 learned_addr learned my addr 172.21.15.86:0/2102309607 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabab7fe700 1 -- 172.21.15.86:0/2102309607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0feeb0 con 0x7fabac070350 2023-11-15T05:00:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabab7fe700 1 --2- 172.21.15.86:0/2102309607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fab9c00b5f0 tx=0x7fab9c006ee0).ready entity=mon.0 client_cookie=1b37f523809e6455 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabaa7fc700 1 -- 172.21.15.86:0/2102309607 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab9c00d600 con 0x7fabac070350 2023-11-15T05:00:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabaa7fc700 1 -- 172.21.15.86:0/2102309607 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab9c00dbd0 con 0x7fabac070350 2023-11-15T05:00:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.998+0000 7fabaa7fc700 1 -- 172.21.15.86:0/2102309607 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab9c011c00 con 0x7fabac070350 2023-11-15T05:00:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 -- 172.21.15.86:0/2102309607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 msgr2=0x7fabac0fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 --2- 172.21.15.86:0/2102309607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fab9c00b5f0 tx=0x7fab9c006ee0).stop 2023-11-15T05:00:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 -- 172.21.15.86:0/2102309607 shutdown_connections 2023-11-15T05:00:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 --2- 172.21.15.86:0/2102309607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac0fe9c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 -- 172.21.15.86:0/2102309607 >> 172.21.15.86:0/2102309607 conn(0x7fabac0f63c0 msgr2=0x7fabac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 -- 172.21.15.86:0/2102309607 shutdown_connections 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:37.999+0000 7fabb1f3e700 1 -- 172.21.15.86:0/2102309607 wait complete. 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabb1f3e700 1 Processor -- start 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabb1f3e700 1 -- start start 2023-11-15T05:00:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabb1f3e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabb1f3e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac10af40 con 0x7fabac070350 2023-11-15T05:00:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52000/0 (socket says 172.21.15.86:52000) 2023-11-15T05:00:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.000+0000 7fabab7fe700 1 -- 172.21.15.86:0/848031551 learned_addr learned my addr 172.21.15.86:0/848031551 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fabab7fe700 1 -- 172.21.15.86:0/848031551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c00b200 con 0x7fabac070350 2023-11-15T05:00:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fabab7fe700 1 --2- 172.21.15.86:0/848031551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fab9c002f40 tx=0x7fab9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab9c011690 con 0x7fabac070350 2023-11-15T05:00:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac10b140 con 0x7fabac070350 2023-11-15T05:00:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab9c01b880 con 0x7fabac070350 2023-11-15T05:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab9c01a770 con 0x7fabac070350 2023-11-15T05:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac10b550 con 0x7fabac070350 2023-11-15T05:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fab9c00f070 con 0x7fabac070350 2023-11-15T05:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.001+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fab9c025a50 con 0x7fabac070350 2023-11-15T05:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.002+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac045ad0 con 0x7fabac070350 2023-11-15T05:00:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.004+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fab9c016070 con 0x7fabac070350 2023-11-15T05:00:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.072+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fabac104790 con 0x7fabac070350 2023-11-15T05:00:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.073+0000 7fab93fff700 1 -- 172.21.15.86:0/848031551 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fab9c01bd60 con 0x7fabac070350 2023-11-15T05:00:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:00:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsid": "d8947496-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T05:00:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:00:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:00:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:00:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:00:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 0 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "smithi086" 2023-11-15T05:00:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T05:00:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:00:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:00:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:00:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:00:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:00:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:00:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:00:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:00:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:00:38.092 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:00:38.092 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:00:38.092 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:00:38.092 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.092 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:00:38.093 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "available": false, 2023-11-15T05:00:38.094 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:00:38.094 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:00:38.094 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:00:38.094 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:00:38.094 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:00:33.110627+0000", 2023-11-15T05:00:38.095 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:38.096 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:38.096 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:00:38.096 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:00:38.096 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 msgr2=0x7fabac10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:38.096 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 --2- 172.21.15.86:0/848031551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fab9c002f40 tx=0x7fab9c006ee0).stop 2023-11-15T05:00:38.097 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 shutdown_connections 2023-11-15T05:00:38.097 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 --2- 172.21.15.86:0/848031551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabac070350 0x7fabac10aa50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:38.097 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 >> 172.21.15.86:0/848031551 conn(0x7fabac0f63c0 msgr2=0x7fabac0f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:38.097 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 shutdown_connections 2023-11-15T05:00:38.097 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:38.074+0000 7fabb1f3e700 1 -- 172.21.15.86:0/848031551 wait complete. 2023-11-15T05:00:38.452 INFO:teuthology.orchestra.run.smithi086.stderr:mgr not available, waiting (1/15)... 2023-11-15T05:00:38.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:38.074551+0000 mon.smithi086 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.86:0/848031551' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:00:40.455 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:00:40.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.823+0000 7f8ef678f700 1 Processor -- start 2023-11-15T05:00:40.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.823+0000 7f8ef678f700 1 -- start start 2023-11-15T05:00:40.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.823+0000 7f8ef678f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:40.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.823+0000 7f8ef678f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00fc750 con 0x7f8ef00703b0 2023-11-15T05:00:40.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:40.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40722/0 (socket says 172.21.15.86:40722) 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeffff700 1 -- 172.21.15.86:0/702156246 learned_addr learned my addr 172.21.15.86:0/702156246 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeffff700 1 -- 172.21.15.86:0/702156246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0101250 con 0x7f8ef00703b0 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeffff700 1 --2- 172.21.15.86:0/702156246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8ee000b5f0 tx=0x7f8ee0006ee0).ready entity=mon.0 client_cookie=2c1be8f4884724b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeeffd700 1 -- 172.21.15.86:0/702156246 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee000d600 con 0x7f8ef00703b0 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeeffd700 1 -- 172.21.15.86:0/702156246 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8ee000dbd0 con 0x7f8ef00703b0 2023-11-15T05:00:40.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.824+0000 7f8eeeffd700 1 -- 172.21.15.86:0/702156246 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee0011c00 con 0x7f8ef00703b0 2023-11-15T05:00:40.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 -- 172.21.15.86:0/702156246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 msgr2=0x7f8ef0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:40.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 --2- 172.21.15.86:0/702156246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8ee000b5f0 tx=0x7f8ee0006ee0).stop 2023-11-15T05:00:40.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 -- 172.21.15.86:0/702156246 shutdown_connections 2023-11-15T05:00:40.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 --2- 172.21.15.86:0/702156246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef0100d60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:40.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 -- 172.21.15.86:0/702156246 >> 172.21.15.86:0/702156246 conn(0x7f8ef00f63c0 msgr2=0x7f8ef00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 -- 172.21.15.86:0/702156246 shutdown_connections 2023-11-15T05:00:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.825+0000 7f8ef678f700 1 -- 172.21.15.86:0/702156246 wait complete. 2023-11-15T05:00:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8ef678f700 1 Processor -- start 2023-11-15T05:00:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8ef678f700 1 -- start start 2023-11-15T05:00:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8ef678f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8ef678f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef010d100 con 0x7f8ef00703b0 2023-11-15T05:00:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8eeffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40724/0 (socket says 172.21.15.86:40724) 2023-11-15T05:00:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.826+0000 7f8eeffff700 1 -- 172.21.15.86:0/4249600357 learned_addr learned my addr 172.21.15.86:0/4249600357 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eeffff700 1 -- 172.21.15.86:0/4249600357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee000b200 con 0x7f8ef00703b0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eeffff700 1 --2- 172.21.15.86:0/4249600357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8ee000b190 tx=0x7f8ee0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee001b410 con 0x7f8ef00703b0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef010d300 con 0x7f8ef00703b0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8ee001b9e0 con 0x7f8ef00703b0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef010d710 con 0x7f8ef00703b0 2023-11-15T05:00:40.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee001a880 con 0x7f8ef00703b0 2023-11-15T05:00:40.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8ee000f070 con 0x7f8ef00703b0 2023-11-15T05:00:40.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.827+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8ee0025b40 con 0x7f8ef00703b0 2023-11-15T05:00:40.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.828+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0004fa0 con 0x7f8ef00703b0 2023-11-15T05:00:40.837 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.830+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8ee0016070 con 0x7f8ef00703b0 2023-11-15T05:00:40.905 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.899+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8ed0005670 con 0x7f8ef00703b0 2023-11-15T05:00:40.906 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.899+0000 7f8eecff9700 1 -- 172.21.15.86:0/4249600357 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f8ee0025d80 con 0x7f8ef00703b0 2023-11-15T05:00:40.906 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.906 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:00:40.907 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsid": "d8947496-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T05:00:40.907 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:00:40.907 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:00:40.907 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:00:40.907 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:00:40.908 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.908 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:00:40.910 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:00:40.910 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 0 2023-11-15T05:00:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:00:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "smithi086" 2023-11-15T05:00:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T05:00:40.912 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:00:40.912 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:40.912 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:00:40.912 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:00:40.912 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.913 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:00:40.913 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:40.913 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:00:40.913 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:00:40.913 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:00:40.914 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:00:40.914 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:00:40.914 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:00:40.914 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.914 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:00:40.915 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:00:40.916 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:00:40.916 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:00:40.916 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.916 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:00:40.916 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "available": false, 2023-11-15T05:00:40.917 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:00:40.918 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:00:40.918 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:00:40.918 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:00:40.918 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:40.918 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:40.919 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.919 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:00:40.919 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:40.919 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:00:33.110627+0000", 2023-11-15T05:00:40.920 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:40.920 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:40.920 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:00:40.922 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:00:40.922 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 msgr2=0x7f8ef010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:40.922 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 --2- 172.21.15.86:0/4249600357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8ee000b190 tx=0x7f8ee0006ee0).stop 2023-11-15T05:00:40.922 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 shutdown_connections 2023-11-15T05:00:40.923 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 --2- 172.21.15.86:0/4249600357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ef00703b0 0x7f8ef010cc10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:40.923 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 >> 172.21.15.86:0/4249600357 conn(0x7f8ef00f63c0 msgr2=0x7f8ef00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:40.923 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 shutdown_connections 2023-11-15T05:00:40.925 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:40.901+0000 7f8ef678f700 1 -- 172.21.15.86:0/4249600357 wait complete. 2023-11-15T05:00:41.204 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:40.900764+0000 mon.smithi086 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.86:0/4249600357' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:00:41.242 INFO:teuthology.orchestra.run.smithi086.stderr:mgr not available, waiting (2/15)... 2023-11-15T05:00:42.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:42.497205+0000 mon.smithi086 (mon.0) 9 : cluster [INF] Activating manager daemon smithi086.hkfamo 2023-11-15T05:00:42.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:42.499961+0000 mon.smithi086 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi086.hkfamo(active, starting, since 0.00286072s) 2023-11-15T05:00:42.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.500917+0000 mon.smithi086 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:00:42.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.501105+0000 mon.smithi086 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:00:42.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.501283+0000 mon.smithi086 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:00:42.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.501453+0000 mon.smithi086 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2023-11-15T05:00:42.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.501633+0000 mon.smithi086 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mgr metadata", "who": "smithi086.hkfamo", "id": "smithi086.hkfamo"}]: dispatch 2023-11-15T05:00:42.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:42.503139+0000 mon.smithi086 (mon.0) 16 : cluster [INF] Manager daemon smithi086.hkfamo is now available 2023-11-15T05:00:42.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.511026+0000 mon.smithi086 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:42.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.514598+0000 mon.smithi086 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:00:42.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.515782+0000 mon.smithi086 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:00:42.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.520466+0000 mon.smithi086 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:42.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:42.522727+0000 mon.smithi086 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.86:0/1372083181' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:43.245 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:00:43.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.603+0000 7fab3af9c700 1 Processor -- start 2023-11-15T05:00:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.603+0000 7fab3af9c700 1 -- start start 2023-11-15T05:00:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab3af9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab3af9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340fc730 con 0x7fab340703d0 2023-11-15T05:00:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab38d38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab38d38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40768/0 (socket says 172.21.15.86:40768) 2023-11-15T05:00:43.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab38d38700 1 -- 172.21.15.86:0/4272086167 learned_addr learned my addr 172.21.15.86:0/4272086167 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:43.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab38d38700 1 -- 172.21.15.86:0/4272086167 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34101230 con 0x7fab340703d0 2023-11-15T05:00:43.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab38d38700 1 --2- 172.21.15.86:0/4272086167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fab2800b5f0 tx=0x7fab28006ee0).ready entity=mon.0 client_cookie=5a1cfc5c6185ab45 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:43.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab337fe700 1 -- 172.21.15.86:0/4272086167 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2800d600 con 0x7fab340703d0 2023-11-15T05:00:43.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab337fe700 1 -- 172.21.15.86:0/4272086167 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab2800dbd0 con 0x7fab340703d0 2023-11-15T05:00:43.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.604+0000 7fab337fe700 1 -- 172.21.15.86:0/4272086167 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab28011c00 con 0x7fab340703d0 2023-11-15T05:00:43.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 -- 172.21.15.86:0/4272086167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 msgr2=0x7fab34100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 --2- 172.21.15.86:0/4272086167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fab2800b5f0 tx=0x7fab28006ee0).stop 2023-11-15T05:00:43.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 -- 172.21.15.86:0/4272086167 shutdown_connections 2023-11-15T05:00:43.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 --2- 172.21.15.86:0/4272086167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab34100d40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 -- 172.21.15.86:0/4272086167 >> 172.21.15.86:0/4272086167 conn(0x7fab340f63c0 msgr2=0x7fab340f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 -- 172.21.15.86:0/4272086167 shutdown_connections 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 -- 172.21.15.86:0/4272086167 wait complete. 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.605+0000 7fab3af9c700 1 Processor -- start 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab3af9c700 1 -- start start 2023-11-15T05:00:43.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab3af9c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab3af9c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3410d0a0 con 0x7fab340703d0 2023-11-15T05:00:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab38d38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab38d38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40780/0 (socket says 172.21.15.86:40780) 2023-11-15T05:00:43.613 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab38d38700 1 -- 172.21.15.86:0/3374328968 learned_addr learned my addr 172.21.15.86:0/3374328968 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:43.615 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab38d38700 1 -- 172.21.15.86:0/3374328968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2800b200 con 0x7fab340703d0 2023-11-15T05:00:43.615 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab38d38700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fab2800b190 tx=0x7fab28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:43.615 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.606+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2801b410 con 0x7fab340703d0 2023-11-15T05:00:43.615 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab2801b9e0 con 0x7fab340703d0 2023-11-15T05:00:43.615 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3410d2a0 con 0x7fab340703d0 2023-11-15T05:00:43.616 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab2801a880 con 0x7fab340703d0 2023-11-15T05:00:43.616 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3410d6b0 con 0x7fab340703d0 2023-11-15T05:00:43.616 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fab2800f070 con 0x7fab340703d0 2023-11-15T05:00:43.616 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.607+0000 7fab317fa700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 0x7fab18032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:43.617 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.608+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fab28041170 con 0x7fab340703d0 2023-11-15T05:00:43.617 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.608+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab34045ad0 con 0x7fab340703d0 2023-11-15T05:00:43.617 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.608+0000 7fab33fff700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 0x7fab18032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:43.617 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.608+0000 7fab33fff700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 0x7fab18032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fab200096a0 tx=0x7fab20006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:43.618 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.612+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab28016070 con 0x7fab340703d0 2023-11-15T05:00:43.822 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.815+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fab34101d90 con 0x7fab340703d0 2023-11-15T05:00:43.822 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.815+0000 7fab317fa700 1 -- 172.21.15.86:0/3374328968 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fab34101d90 con 0x7fab340703d0 2023-11-15T05:00:43.822 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.823 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:00:43.823 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsid": "d8947496-8373-11ee-8dbe-212e2dc638e7", 2023-11-15T05:00:43.823 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:00:43.823 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:00:43.823 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:00:43.824 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:00:43.824 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.824 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:00:43.824 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:00:43.824 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 0 2023-11-15T05:00:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:00:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "smithi086" 2023-11-15T05:00:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T05:00:43.826 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:00:43.826 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:43.826 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:00:43.826 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:00:43.826 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.827 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:00:43.827 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:43.827 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:00:43.827 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:00:43.827 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:00:43.828 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:00:43.828 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:00:43.828 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:00:43.828 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:00:43.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:00:43.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:00:43.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:00:43.829 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:00:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:00:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:00:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:00:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:00:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:00:43.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:43.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:00:43.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:00:43.831 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:00:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:00:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:00:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:00:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ], 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:00:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:00:43.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:00:33.110627+0000", 2023-11-15T05:00:43.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:00:43.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: }, 2023-11-15T05:00:43.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:00:43.834 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:00:43.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 msgr2=0x7fab18032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 0x7fab18032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fab200096a0 tx=0x7fab20006b40).stop 2023-11-15T05:00:43.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 msgr2=0x7fab3410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:43.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fab2800b190 tx=0x7fab28006ee0).stop 2023-11-15T05:00:43.835 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 shutdown_connections 2023-11-15T05:00:43.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7fab1802fc50 0x7fab18032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 --2- 172.21.15.86:0/3374328968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab340703d0 0x7fab3410cbb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:43.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 >> 172.21.15.86:0/3374328968 conn(0x7fab340f63c0 msgr2=0x7fab340f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:43.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 shutdown_connections 2023-11-15T05:00:43.836 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:43.817+0000 7fab3af9c700 1 -- 172.21.15.86:0/3374328968 wait complete. 2023-11-15T05:00:44.168 INFO:teuthology.orchestra.run.smithi086.stderr:mgr is available 2023-11-15T05:00:44.168 INFO:teuthology.orchestra.run.smithi086.stderr:Enabling cephadm module... 2023-11-15T05:00:44.168 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T05:00:44.512 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.509+0000 7ff367b26700 1 Processor -- start 2023-11-15T05:00:44.512 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.509+0000 7ff367b26700 1 -- start start 2023-11-15T05:00:44.512 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.509+0000 7ff367b26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:44.513 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.509+0000 7ff367b26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600fee10 con 0x7ff3600703f0 2023-11-15T05:00:44.513 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3658c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:44.513 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3658c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40784/0 (socket says 172.21.15.86:40784) 2023-11-15T05:00:44.513 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3658c2700 1 -- 172.21.15.86:0/1939814933 learned_addr learned my addr 172.21.15.86:0/1939814933 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:44.513 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3658c2700 1 -- 172.21.15.86:0/1939814933 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600fef50 con 0x7ff3600703f0 2023-11-15T05:00:44.514 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3658c2700 1 --2- 172.21.15.86:0/1939814933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff35000b5f0 tx=0x7ff350006ee0).ready entity=mon.0 client_cookie=ced47f0c867383c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:44.514 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3648c0700 1 -- 172.21.15.86:0/1939814933 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35000d600 con 0x7ff3600703f0 2023-11-15T05:00:44.514 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3648c0700 1 -- 172.21.15.86:0/1939814933 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff35000dbd0 con 0x7ff3600703f0 2023-11-15T05:00:44.514 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.510+0000 7ff3648c0700 1 -- 172.21.15.86:0/1939814933 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff350011c00 con 0x7ff3600703f0 2023-11-15T05:00:44.514 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- 172.21.15.86:0/1939814933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 msgr2=0x7ff3600fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:44.515 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 --2- 172.21.15.86:0/1939814933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff35000b5f0 tx=0x7ff350006ee0).stop 2023-11-15T05:00:44.515 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- 172.21.15.86:0/1939814933 shutdown_connections 2023-11-15T05:00:44.515 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 --2- 172.21.15.86:0/1939814933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fe920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:44.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- 172.21.15.86:0/1939814933 >> 172.21.15.86:0/1939814933 conn(0x7ff3600f63c0 msgr2=0x7ff3600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:44.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- 172.21.15.86:0/1939814933 shutdown_connections 2023-11-15T05:00:44.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- 172.21.15.86:0/1939814933 wait complete. 2023-11-15T05:00:44.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 Processor -- start 2023-11-15T05:00:44.519 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- start start 2023-11-15T05:00:44.519 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:44.519 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff367b26700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3600fa820 con 0x7ff3600703f0 2023-11-15T05:00:44.519 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.511+0000 7ff3658c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:44.519 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff3658c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40798/0 (socket says 172.21.15.86:40798) 2023-11-15T05:00:44.520 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff3658c2700 1 -- 172.21.15.86:0/1915874469 learned_addr learned my addr 172.21.15.86:0/1915874469 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:44.520 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff3658c2700 1 -- 172.21.15.86:0/1915874469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35000b200 con 0x7ff3600703f0 2023-11-15T05:00:44.520 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff3658c2700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff35000b190 tx=0x7ff350006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:44.520 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35001b410 con 0x7ff3600703f0 2023-11-15T05:00:44.520 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff35001b9e0 con 0x7ff3600703f0 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3600faa20 con 0x7ff3600703f0 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35001a880 con 0x7ff3600703f0 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.512+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3600fae30 con 0x7ff3600703f0 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7ff35000f070 con 0x7ff3600703f0 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff34e7fc700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 0x7ff348032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:44.521 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff350040fc0 con 0x7ff3600703f0 2023-11-15T05:00:44.522 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360045ad0 con 0x7ff3600703f0 2023-11-15T05:00:44.522 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff3650c1700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 0x7ff348032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:44.522 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.513+0000 7ff3650c1700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 0x7ff348032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff35c0096a0 tx=0x7ff35c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:44.523 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.517+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3500224f0 con 0x7ff3600703f0 2023-11-15T05:00:44.702 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:44.701+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff3600fb130 con 0x7ff3600703f0 2023-11-15T05:00:44.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:43.502852+0000 mon.smithi086 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi086.hkfamo(active, since 1.00575s) 2023-11-15T05:00:44.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:43.816662+0000 mon.smithi086 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.86:0/3374328968' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:00:45.502 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.500+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff350025460 con 0x7ff3600703f0 2023-11-15T05:00:45.503 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.502+0000 7ff34e7fc700 1 -- 172.21.15.86:0/1915874469 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff350024d50 con 0x7ff3600703f0 2023-11-15T05:00:45.507 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 msgr2=0x7ff348032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:45.507 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 0x7ff348032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff35c0096a0 tx=0x7ff35c006b40).stop 2023-11-15T05:00:45.507 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 msgr2=0x7ff3600fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:45.507 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ff35000b190 tx=0x7ff350006ee0).stop 2023-11-15T05:00:45.507 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 shutdown_connections 2023-11-15T05:00:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7ff34802fca0 0x7ff348032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 --2- 172.21.15.86:0/1915874469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3600703f0 0x7ff3600fc330 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.504+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 >> 172.21.15.86:0/1915874469 conn(0x7ff3600f63c0 msgr2=0x7ff3600f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.505+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 shutdown_connections 2023-11-15T05:00:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:45.505+0000 7ff367b26700 1 -- 172.21.15.86:0/1915874469 wait complete. 2023-11-15T05:00:45.812 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:44.702620+0000 mon.smithi086 (mon.0) 24 : audit [INF] from='client.? 172.21.15.86:0/1915874469' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T05:00:45.876 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T05:00:46.225 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f5620069700 1 Processor -- start 2023-11-15T05:00:46.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f5620069700 1 -- start start 2023-11-15T05:00:46.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f5620069700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:46.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f5620069700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56180fc710 con 0x7f56180703f0 2023-11-15T05:00:46.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f561de05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:46.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.223+0000 7f561de05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40826/0 (socket says 172.21.15.86:40826) 2023-11-15T05:00:46.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561de05700 1 -- 172.21.15.86:0/1320935311 learned_addr learned my addr 172.21.15.86:0/1320935311 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:46.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561de05700 1 -- 172.21.15.86:0/1320935311 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5618101210 con 0x7f56180703f0 2023-11-15T05:00:46.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561de05700 1 --2- 172.21.15.86:0/1320935311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f561400b5f0 tx=0x7f5614006ee0).ready entity=mon.0 client_cookie=722cb159dfae0f60 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:46.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561ce03700 1 -- 172.21.15.86:0/1320935311 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f561400d600 con 0x7f56180703f0 2023-11-15T05:00:46.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561ce03700 1 -- 172.21.15.86:0/1320935311 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f561400dbd0 con 0x7f56180703f0 2023-11-15T05:00:46.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.224+0000 7f561ce03700 1 -- 172.21.15.86:0/1320935311 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5614011c00 con 0x7f56180703f0 2023-11-15T05:00:46.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 -- 172.21.15.86:0/1320935311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 msgr2=0x7f5618100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:46.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 --2- 172.21.15.86:0/1320935311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f561400b5f0 tx=0x7f5614006ee0).stop 2023-11-15T05:00:46.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 -- 172.21.15.86:0/1320935311 shutdown_connections 2023-11-15T05:00:46.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 --2- 172.21.15.86:0/1320935311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f5618100d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:46.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 -- 172.21.15.86:0/1320935311 >> 172.21.15.86:0/1320935311 conn(0x7f56180f63c0 msgr2=0x7f56180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:46.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 -- 172.21.15.86:0/1320935311 shutdown_connections 2023-11-15T05:00:46.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.225+0000 7f5620069700 1 -- 172.21.15.86:0/1320935311 wait complete. 2023-11-15T05:00:46.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f5620069700 1 Processor -- start 2023-11-15T05:00:46.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f5620069700 1 -- start start 2023-11-15T05:00:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f5620069700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f5620069700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561810d1e0 con 0x7f56180703f0 2023-11-15T05:00:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f561de05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f561de05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40836/0 (socket says 172.21.15.86:40836) 2023-11-15T05:00:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f561de05700 1 -- 172.21.15.86:0/4258273816 learned_addr learned my addr 172.21.15.86:0/4258273816 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.226+0000 7f561de05700 1 -- 172.21.15.86:0/4258273816 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561400b200 con 0x7f56180703f0 2023-11-15T05:00:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f561de05700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f561400e040 tx=0x7f5614006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f561400d9f0 con 0x7f56180703f0 2023-11-15T05:00:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561810d3e0 con 0x7f56180703f0 2023-11-15T05:00:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f561401a9f0 con 0x7f56180703f0 2023-11-15T05:00:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f561401b8c0 con 0x7f56180703f0 2023-11-15T05:00:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.227+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561810d7f0 con 0x7f56180703f0 2023-11-15T05:00:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f561400f070 con 0x7f56180703f0 2023-11-15T05:00:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f56067fc700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 0x7f5600032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:46.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5614041250 con 0x7f56180703f0 2023-11-15T05:00:46.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f561d604700 1 -- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 msgr2=0x7f5600032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:46.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f561d604700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 0x7f5600032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:00:46.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.228+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5618045ad0 con 0x7f56180703f0 2023-11-15T05:00:46.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.232+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5614016070 con 0x7f56180703f0 2023-11-15T05:00:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.429+0000 7f561d604700 1 -- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 msgr2=0x7f5600032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.429+0000 7f561d604700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 0x7f5600032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:00:46.454 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.452+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f561810ece0 con 0x7f56180703f0 2023-11-15T05:00:46.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.453+0000 7f56067fc700 1 -- 172.21.15.86:0/4258273816 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f5614025460 con 0x7f56180703f0 2023-11-15T05:00:46.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:00:46.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 4, 2023-11-15T05:00:46.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:00:46.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "active_name": "smithi086.hkfamo", 2023-11-15T05:00:46.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T05:00:46.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:00:46.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.454+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 msgr2=0x7f5600032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:00:46.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.454+0000 7f5620069700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 0x7f5600032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:00:46.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.454+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 msgr2=0x7f561810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:46.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.454+0000 7f5620069700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f561400e040 tx=0x7f5614006ee0).stop 2023-11-15T05:00:46.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.454+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 shutdown_connections 2023-11-15T05:00:46.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.455+0000 7f5620069700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f560002fd50 0x7f5600032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:00:46.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.455+0000 7f5620069700 1 --2- 172.21.15.86:0/4258273816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56180703f0 0x7f561810ccf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:46.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.455+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 >> 172.21.15.86:0/4258273816 conn(0x7f56180f63c0 msgr2=0x7f56180f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:46.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.455+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 shutdown_connections 2023-11-15T05:00:46.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:46.455+0000 7f5620069700 1 -- 172.21.15.86:0/4258273816 wait complete. 2023-11-15T05:00:46.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:46 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:45.503637+0000 mon.smithi086 (mon.0) 25 : audit [INF] from='client.? 172.21.15.86:0/1915874469' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T05:00:46.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:46 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:45.503754+0000 mon.smithi086 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi086.hkfamo(active, since 3s) 2023-11-15T05:00:46.736 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:46 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:46.454524+0000 mon.smithi086 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.86:0/4258273816' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T05:00:46.795 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for the mgr to restart... 2023-11-15T05:00:46.795 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mgr epoch 4... 2023-11-15T05:00:46.795 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T05:00:47.183 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.180+0000 7f0cdad17700 1 Processor -- start 2023-11-15T05:00:47.184 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.180+0000 7f0cdad17700 1 -- start start 2023-11-15T05:00:47.184 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.180+0000 7f0cdad17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:47.184 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.180+0000 7f0cdad17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40fee10 con 0x7f0cd40703f0 2023-11-15T05:00:47.184 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.180+0000 7f0cd8ab3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:47.184 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd8ab3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40840/0 (socket says 172.21.15.86:40840) 2023-11-15T05:00:47.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd8ab3700 1 -- 172.21.15.86:0/27159993 learned_addr learned my addr 172.21.15.86:0/27159993 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:47.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd8ab3700 1 -- 172.21.15.86:0/27159993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40fef50 con 0x7f0cd40703f0 2023-11-15T05:00:47.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd8ab3700 1 --2- 172.21.15.86:0/27159993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0cc800b5f0 tx=0x7f0cc8006ee0).ready entity=mon.0 client_cookie=e729bc08ad4c7335 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:47.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd37fe700 1 -- 172.21.15.86:0/27159993 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc800d600 con 0x7f0cd40703f0 2023-11-15T05:00:47.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd37fe700 1 -- 172.21.15.86:0/27159993 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cc800dbd0 con 0x7f0cd40703f0 2023-11-15T05:00:47.186 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.181+0000 7f0cd37fe700 1 -- 172.21.15.86:0/27159993 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc8011c00 con 0x7f0cd40703f0 2023-11-15T05:00:47.186 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 -- 172.21.15.86:0/27159993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 msgr2=0x7f0cd40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:47.186 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 --2- 172.21.15.86:0/27159993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0cc800b5f0 tx=0x7f0cc8006ee0).stop 2023-11-15T05:00:47.186 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 -- 172.21.15.86:0/27159993 shutdown_connections 2023-11-15T05:00:47.186 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 --2- 172.21.15.86:0/27159993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd40fe920 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:47.187 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 -- 172.21.15.86:0/27159993 >> 172.21.15.86:0/27159993 conn(0x7f0cd40f63c0 msgr2=0x7f0cd40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:47.187 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 -- 172.21.15.86:0/27159993 shutdown_connections 2023-11-15T05:00:47.187 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.182+0000 7f0cdad17700 1 -- 172.21.15.86:0/27159993 wait complete. 2023-11-15T05:00:47.189 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cdad17700 1 Processor -- start 2023-11-15T05:00:47.189 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cdad17700 1 -- start start 2023-11-15T05:00:47.189 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cdad17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:47.190 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cdad17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd410af40 con 0x7f0cd40703f0 2023-11-15T05:00:47.190 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cd8ab3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:47.190 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cd8ab3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40848/0 (socket says 172.21.15.86:40848) 2023-11-15T05:00:47.190 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.183+0000 7f0cd8ab3700 1 -- 172.21.15.86:0/990273325 learned_addr learned my addr 172.21.15.86:0/990273325 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:47.190 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cd8ab3700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc800b200 con 0x7f0cd40703f0 2023-11-15T05:00:47.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cd8ab3700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0cc800e040 tx=0x7f0cc8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:47.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc800d9f0 con 0x7f0cd40703f0 2023-11-15T05:00:47.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cc801a9f0 con 0x7f0cd40703f0 2023-11-15T05:00:47.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cc801b8c0 con 0x7f0cd40703f0 2023-11-15T05:00:47.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd410b140 con 0x7f0cd40703f0 2023-11-15T05:00:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.184+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd410b550 con 0x7f0cd40703f0 2023-11-15T05:00:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0cc800f070 con 0x7f0cd40703f0 2023-11-15T05:00:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cd17fa700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cd4059100 con 0x7f0cb802fd50 2023-11-15T05:00:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0cc8041270 con 0x7f0cd40703f0 2023-11-15T05:00:47.193 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cd3fff700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:47.193 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.185+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:00:47.387 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.386+0000 7f0cd3fff700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:47.388 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.386+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:00:47.788 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.787+0000 7f0cd3fff700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:47.788 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:47.787+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:00:48.589 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:48.588+0000 7f0cd3fff700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:48.590 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:48.588+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:00:50.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:50.189+0000 7f0cd3fff700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/711176615 2023-11-15T05:00:50.191 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:50.189+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T05:00:50.501 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:50.500+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f0cc80257e0 con 0x7f0cd40703f0 2023-11-15T05:00:50.501 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:50.500+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 msgr2=0x7f0cb8032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:00:50.502 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:50.500+0000 7f0cd17fa700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:00:50.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:50.449217+0000 mon.smithi086 (mon.0) 28 : cluster [INF] Active manager daemon smithi086.hkfamo restarted 2023-11-15T05:00:50.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:50.449643+0000 mon.smithi086 (mon.0) 29 : cluster [INF] Activating manager daemon smithi086.hkfamo 2023-11-15T05:00:50.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:50.451424+0000 mon.smithi086 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T05:00:51.517 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.515+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f0cc8042850 con 0x7f0cd40703f0 2023-11-15T05:00:51.517 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.515+0000 7f0cd17fa700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 0x7f0cb8035ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:51.517 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.515+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cd4059100 con 0x7f0cb80336c0 2023-11-15T05:00:51.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.515+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 0x7f0cb8035ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:51.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.516+0000 7f0cd3fff700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 0x7f0cb8035ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0cc0006fd0 tx=0x7f0cc0006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:51.518 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.517+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f0cd4059100 con 0x7f0cb80336c0 2023-11-15T05:00:51.525 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.523+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0cd4045ad0 con 0x7f0cb80336c0 2023-11-15T05:00:51.525 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.523+0000 7f0cd17fa700 1 -- 172.21.15.86:0/990273325 <== mgr.14114 v2:172.21.15.86:6800/2368880557 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0cd4045ad0 con 0x7f0cb80336c0 2023-11-15T05:00:51.525 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:00:51.526 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-11-15T05:00:51.526 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T05:00:51.526 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:00:51.527 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 msgr2=0x7f0cb8035ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:51.527 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 0x7f0cb8035ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0cc0006fd0 tx=0x7f0cc0006cf0).stop 2023-11-15T05:00:51.527 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 msgr2=0x7f0cd410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:51.527 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0cc800e040 tx=0x7f0cc8006ee0).stop 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 shutdown_connections 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd40703f0 0x7f0cd410aa50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cb80336c0 0x7f0cb8035ab0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 --2- 172.21.15.86:0/990273325 >> [v2:172.21.15.86:6800/711176615,v1:172.21.15.86:6801/711176615] conn(0x7f0cb802fd50 0x7f0cb8032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 >> 172.21.15.86:0/990273325 conn(0x7f0cd40f63c0 msgr2=0x7f0cd40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:51.528 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 shutdown_connections 2023-11-15T05:00:51.529 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:51.524+0000 7f0cdad17700 1 -- 172.21.15.86:0/990273325 wait complete. 2023-11-15T05:00:51.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:50.502073+0000 mon.smithi086 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi086.hkfamo(active, starting, since 0.0525062s) 2023-11-15T05:00:51.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.504358+0000 mon.smithi086 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2023-11-15T05:00:51.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.504963+0000 mon.smithi086 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mgr metadata", "who": "smithi086.hkfamo", "id": "smithi086.hkfamo"}]: dispatch 2023-11-15T05:00:51.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.505621+0000 mon.smithi086 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:00:51.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.505806+0000 mon.smithi086 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:00:51.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.505972+0000 mon.smithi086 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:00:51.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:50.519669+0000 mon.smithi086 (mon.0) 37 : cluster [INF] Manager daemon smithi086.hkfamo is now available 2023-11-15T05:00:51.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.535896+0000 mon.smithi086 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:51.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.541422+0000 mon.smithi086 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:51.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.546753+0000 mon.smithi086 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:51.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.636372+0000 mon.smithi086 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:51.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.637022+0000 mon.smithi086 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:00:51.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.638601+0000 mon.smithi086 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:00:51.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.639982+0000 mon.smithi086 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:00:51.992 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.649584+0000 mon.smithi086 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:00:51.993 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:50.650746+0000 mon.smithi086 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:00:52.794 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:51.518165+0000 mgr.smithi086.hkfamo (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T05:00:52.794 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:51.518746+0000 mon.smithi086 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi086.hkfamo(active, since 1.06917s) 2023-11-15T05:00:52.794 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:51.524925+0000 mgr.smithi086.hkfamo (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T05:00:53.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:00:52.521474+0000 mon.smithi086 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi086.hkfamo(active, since 2s) 2023-11-15T05:00:54.017 INFO:teuthology.orchestra.run.smithi086.stderr:mgr epoch 4 is available 2023-11-15T05:00:54.017 INFO:teuthology.orchestra.run.smithi086.stderr:Setting orchestrator backend to cephadm... 2023-11-15T05:00:54.017 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T05:00:55.426 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91b1971700 1 Processor -- start 2023-11-15T05:00:55.427 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91b1971700 1 -- start start 2023-11-15T05:00:55.427 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91b1971700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:55.427 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91b1971700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac072940 con 0x7f91ac072520 2023-11-15T05:00:55.427 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91aaffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:55.428 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91aaffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42514/0 (socket says 172.21.15.86:42514) 2023-11-15T05:00:55.428 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91aaffd700 1 -- 172.21.15.86:0/640922260 learned_addr learned my addr 172.21.15.86:0/640922260 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:55.428 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91aaffd700 1 -- 172.21.15.86:0/640922260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac070fb0 con 0x7f91ac072520 2023-11-15T05:00:55.428 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.368+0000 7f91aaffd700 1 --2- 172.21.15.86:0/640922260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f919c00b5f0 tx=0x7f919c006ee0).ready entity=mon.0 client_cookie=4e922f8d84987e98 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:55.428 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91a9ffb700 1 -- 172.21.15.86:0/640922260 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919c00d600 con 0x7f91ac072520 2023-11-15T05:00:55.429 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91a9ffb700 1 -- 172.21.15.86:0/640922260 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f919c00dbd0 con 0x7f91ac072520 2023-11-15T05:00:55.429 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91a9ffb700 1 -- 172.21.15.86:0/640922260 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919c011c00 con 0x7f91ac072520 2023-11-15T05:00:55.429 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 -- 172.21.15.86:0/640922260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 msgr2=0x7f91ac070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:55.429 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 --2- 172.21.15.86:0/640922260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f919c00b5f0 tx=0x7f919c006ee0).stop 2023-11-15T05:00:55.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 -- 172.21.15.86:0/640922260 shutdown_connections 2023-11-15T05:00:55.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 --2- 172.21.15.86:0/640922260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac070ac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:55.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 -- 172.21.15.86:0/640922260 >> 172.21.15.86:0/640922260 conn(0x7f91ac0f74b0 msgr2=0x7f91ac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:55.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 -- 172.21.15.86:0/640922260 shutdown_connections 2023-11-15T05:00:55.430 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.369+0000 7f91b1971700 1 -- 172.21.15.86:0/640922260 wait complete. 2023-11-15T05:00:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91b1971700 1 Processor -- start 2023-11-15T05:00:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91b1971700 1 -- start start 2023-11-15T05:00:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91b1971700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91b1971700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac10d200 con 0x7f91ac072520 2023-11-15T05:00:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91aaffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91aaffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42522/0 (socket says 172.21.15.86:42522) 2023-11-15T05:00:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91aaffd700 1 -- 172.21.15.86:0/993755359 learned_addr learned my addr 172.21.15.86:0/993755359 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91aaffd700 1 -- 172.21.15.86:0/993755359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c00b200 con 0x7f91ac072520 2023-11-15T05:00:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.370+0000 7f91aaffd700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f919c00b1d0 tx=0x7f919c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919c011a20 con 0x7f91ac072520 2023-11-15T05:00:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f919c01b880 con 0x7f91ac072520 2023-11-15T05:00:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac10d400 con 0x7f91ac072520 2023-11-15T05:00:55.434 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919c01a700 con 0x7f91ac072520 2023-11-15T05:00:55.434 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac10d810 con 0x7f91ac072520 2023-11-15T05:00:55.434 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.371+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f919c00f070 con 0x7f91ac072520 2023-11-15T05:00:55.434 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.372+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac045ad0 con 0x7f91ac072520 2023-11-15T05:00:55.435 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.372+0000 7f91937fe700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 0x7f9198032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:55.435 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.372+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f919c041080 con 0x7f91ac072520 2023-11-15T05:00:55.435 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.372+0000 7f91aa7fc700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 0x7f9198032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:55.435 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.372+0000 7f91aa7fc700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 0x7f9198032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f91a00096a0 tx=0x7f91a0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:55.435 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.376+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f919c022310 con 0x7f91ac072520 2023-11-15T05:00:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.557+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f91ac10df40 con 0x7f919802fd50 2023-11-15T05:00:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.563+0000 7f91937fe700 1 -- 172.21.15.86:0/993755359 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f91ac10df40 con 0x7f919802fd50 2023-11-15T05:00:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 msgr2=0x7f9198032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 0x7f9198032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f91a00096a0 tx=0x7f91a0006b40).stop 2023-11-15T05:00:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 msgr2=0x7f91ac10cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f919c00b1d0 tx=0x7f919c006ee0).stop 2023-11-15T05:00:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 shutdown_connections 2023-11-15T05:00:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f919802fd50 0x7f9198032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:55.569 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 --2- 172.21.15.86:0/993755359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91ac072520 0x7f91ac10cd10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 >> 172.21.15.86:0/993755359 conn(0x7f91ac0f74b0 msgr2=0x7f91ac0f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 shutdown_connections 2023-11-15T05:00:55.570 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:55.565+0000 7f91b1971700 1 -- 172.21.15.86:0/993755359 wait complete. 2023-11-15T05:00:56.931 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:55.560123+0000 mgr.smithi086.hkfamo (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:00:56.932 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:55.563866+0000 mon.smithi086 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:00:56.932 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:55.567172+0000 mon.smithi086 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:00:57.938 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T05:00:59.224 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.129+0000 7f53fbe37700 1 Processor -- start 2023-11-15T05:00:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53fbe37700 1 -- start start 2023-11-15T05:00:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53fbe37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53fbe37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f40fb490 con 0x7f53f40fab80 2023-11-15T05:00:59.225 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53f9bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53f9bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44646/0 (socket says 172.21.15.86:44646) 2023-11-15T05:00:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53f9bd3700 1 -- 172.21.15.86:0/1396121826 learned_addr learned my addr 172.21.15.86:0/1396121826 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53f9bd3700 1 -- 172.21.15.86:0/1396121826 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40fb5d0 con 0x7f53f40fab80 2023-11-15T05:00:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.130+0000 7f53f9bd3700 1 --2- 172.21.15.86:0/1396121826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f53e800b5f0 tx=0x7f53e8006ee0).ready entity=mon.0 client_cookie=359b0744d389233b server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:59.226 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53f8bd1700 1 -- 172.21.15.86:0/1396121826 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e800d600 con 0x7f53f40fab80 2023-11-15T05:00:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53f8bd1700 1 -- 172.21.15.86:0/1396121826 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53e800dbd0 con 0x7f53f40fab80 2023-11-15T05:00:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53f8bd1700 1 -- 172.21.15.86:0/1396121826 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e8011c00 con 0x7f53f40fab80 2023-11-15T05:00:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 -- 172.21.15.86:0/1396121826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 msgr2=0x7f53f40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 --2- 172.21.15.86:0/1396121826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f53e800b5f0 tx=0x7f53e8006ee0).stop 2023-11-15T05:00:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 -- 172.21.15.86:0/1396121826 shutdown_connections 2023-11-15T05:00:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 --2- 172.21.15.86:0/1396121826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f40fafa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 -- 172.21.15.86:0/1396121826 >> 172.21.15.86:0/1396121826 conn(0x7f53f40f63c0 msgr2=0x7f53f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:59.228 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 -- 172.21.15.86:0/1396121826 shutdown_connections 2023-11-15T05:00:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.131+0000 7f53fbe37700 1 -- 172.21.15.86:0/1396121826 wait complete. 2023-11-15T05:00:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53fbe37700 1 Processor -- start 2023-11-15T05:00:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53fbe37700 1 -- start start 2023-11-15T05:00:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53fbe37700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53fbe37700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f410af00 con 0x7f53f40fab80 2023-11-15T05:00:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53f9bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53f9bd3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44652/0 (socket says 172.21.15.86:44652) 2023-11-15T05:00:59.230 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53f9bd3700 1 -- 172.21.15.86:0/720220767 learned_addr learned my addr 172.21.15.86:0/720220767 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:00:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.132+0000 7f53f9bd3700 1 -- 172.21.15.86:0/720220767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e800b200 con 0x7f53f40fab80 2023-11-15T05:00:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53f9bd3700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f53e800b1d0 tx=0x7f53e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e801b410 con 0x7f53f40fab80 2023-11-15T05:00:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f410b100 con 0x7f53f40fab80 2023-11-15T05:00:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53e801b9e0 con 0x7f53f40fab80 2023-11-15T05:00:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e801a840 con 0x7f53f40fab80 2023-11-15T05:00:59.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.133+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f410b510 con 0x7f53f40fab80 2023-11-15T05:00:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f53e800f070 con 0x7f53f40fab80 2023-11-15T05:00:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53e67fc700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 0x7f53dc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:00:59.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f53e80412e0 con 0x7f53f40fab80 2023-11-15T05:00:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53f93d2700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 0x7f53dc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:00:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f4059100 con 0x7f53f40fab80 2023-11-15T05:00:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.134+0000 7f53f93d2700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 0x7f53dc032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:00:59.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.138+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53e8016070 con 0x7f53f40fab80 2023-11-15T05:00:59.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.291+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f53f4104a30 con 0x7f53dc02fd50 2023-11-15T05:00:59.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.292+0000 7f53e67fc700 1 -- 172.21.15.86:0/720220767 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f53f4104a30 con 0x7f53dc02fd50 2023-11-15T05:00:59.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: value unchanged 2023-11-15T05:00:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.293+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 msgr2=0x7f53dc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.293+0000 7f53fbe37700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 0x7f53dc032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f53f00096a0 tx=0x7f53f0006b40).stop 2023-11-15T05:00:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 msgr2=0x7f53f410aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:00:59.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f53e800b1d0 tx=0x7f53e8006ee0).stop 2023-11-15T05:00:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 shutdown_connections 2023-11-15T05:00:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f53dc02fd50 0x7f53dc032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 --2- 172.21.15.86:0/720220767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53f40fab80 0x7f53f410aa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:00:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 >> 172.21.15.86:0/720220767 conn(0x7f53f40f63c0 msgr2=0x7f53f4070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:00:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 shutdown_connections 2023-11-15T05:00:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:00:59.294+0000 7f53fbe37700 1 -- 172.21.15.86:0/720220767 wait complete. 2023-11-15T05:00:59.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:00:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:00:59.293592+0000 mgr.smithi086.hkfamo (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:Generating ssh key... 2023-11-15T05:01:00.875 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T05:01:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.229+0000 7f9cef878700 1 Processor -- start 2023-11-15T05:01:01.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.229+0000 7f9cef878700 1 -- start start 2023-11-15T05:01:01.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.229+0000 7f9cef878700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:01.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.229+0000 7f9cef878700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce80fc5c0 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9ced614700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:01.232 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9ced614700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44656/0 (socket says 172.21.15.86:44656) 2023-11-15T05:01:01.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9ced614700 1 -- 172.21.15.86:0/1172131323 learned_addr learned my addr 172.21.15.86:0/1172131323 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:01.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9ced614700 1 -- 172.21.15.86:0/1172131323 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80fc700 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9ced614700 1 --2- 172.21.15.86:0/1172131323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f9cdc00b5f0 tx=0x7f9cdc006ee0).ready entity=mon.0 client_cookie=2abdf92411681f8c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:01.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9cdbfff700 1 -- 172.21.15.86:0/1172131323 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc00d600 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.233 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9cdbfff700 1 -- 172.21.15.86:0/1172131323 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cdc00dbd0 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.230+0000 7f9cdbfff700 1 -- 172.21.15.86:0/1172131323 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc011c00 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 -- 172.21.15.86:0/1172131323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 msgr2=0x7f9ce80fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 --2- 172.21.15.86:0/1172131323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f9cdc00b5f0 tx=0x7f9cdc006ee0).stop 2023-11-15T05:01:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 -- 172.21.15.86:0/1172131323 shutdown_connections 2023-11-15T05:01:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 --2- 172.21.15.86:0/1172131323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80fc0d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:01.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 -- 172.21.15.86:0/1172131323 >> 172.21.15.86:0/1172131323 conn(0x7f9ce80f74b0 msgr2=0x7f9ce80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:01.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 -- 172.21.15.86:0/1172131323 shutdown_connections 2023-11-15T05:01:01.235 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.231+0000 7f9cef878700 1 -- 172.21.15.86:0/1172131323 wait complete. 2023-11-15T05:01:01.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9cef878700 1 Processor -- start 2023-11-15T05:01:01.237 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9cef878700 1 -- start start 2023-11-15T05:01:01.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9cef878700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:01.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9cef878700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8070a60 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9ced614700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:01.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9ced614700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44662/0 (socket says 172.21.15.86:44662) 2023-11-15T05:01:01.238 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9ced614700 1 -- 172.21.15.86:0/3085233133 learned_addr learned my addr 172.21.15.86:0/3085233133 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:01.239 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9ced614700 1 -- 172.21.15.86:0/3085233133 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc00b200 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.239 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9ced614700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9cdc00b1d0 tx=0x7f9cdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:01.239 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.232+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc01b410 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.239 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.233+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cdc01b9e0 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.239 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.233+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce8070c00 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.240 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.233+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc01a840 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.240 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.233+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce8071010 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.240 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.233+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9cdc00f070 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.240 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.234+0000 7f9cd9ffb700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 0x7f9cd0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:01.240 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.234+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9cdc041220 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.241 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.234+0000 7f9cece13700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 0x7f9cd0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:01.241 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.234+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8059100 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.241 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.234+0000 7f9cece13700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 0x7f9cd0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9ce40096a0 tx=0x7f9ce4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:01.241 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.238+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cdc025460 con 0x7f9ce80fbcb0 2023-11-15T05:01:01.389 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.389+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce80fda30 con 0x7f9cd002fdb0 2023-11-15T05:01:01.756 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.755+0000 7f9cd9ffb700 1 -- 172.21.15.86:0/3085233133 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ce80fda30 con 0x7f9cd002fdb0 2023-11-15T05:01:01.758 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 msgr2=0x7f9cd0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:01.758 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 0x7f9cd0032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9ce40096a0 tx=0x7f9ce4006b40).stop 2023-11-15T05:01:01.758 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 msgr2=0x7f9ce80725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:01.758 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f9cdc00b1d0 tx=0x7f9cdc006ee0).stop 2023-11-15T05:01:01.759 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 shutdown_connections 2023-11-15T05:01:01.759 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd002fdb0 0x7f9cd0032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:01.759 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 --2- 172.21.15.86:0/3085233133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ce80fbcb0 0x7f9ce80725e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:01.759 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 >> 172.21.15.86:0/3085233133 conn(0x7f9ce80f74b0 msgr2=0x7f9ce80f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:01.759 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 shutdown_connections 2023-11-15T05:01:01.760 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:01.757+0000 7f9cef878700 1 -- 172.21.15.86:0/3085233133 wait complete. 2023-11-15T05:01:02.107 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T05:01:02.450 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.444+0000 7f0cdc84b700 1 Processor -- start 2023-11-15T05:01:02.450 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.444+0000 7f0cdc84b700 1 -- start start 2023-11-15T05:01:02.450 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cdc84b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:02.450 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cdc84b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80fc710 con 0x7f0cd80703f0 2023-11-15T05:01:02.451 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:02.451 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44672/0 (socket says 172.21.15.86:44672) 2023-11-15T05:01:02.451 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd659c700 1 -- 172.21.15.86:0/2704300271 learned_addr learned my addr 172.21.15.86:0/2704300271 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:02.451 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd659c700 1 -- 172.21.15.86:0/2704300271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8101210 con 0x7f0cd80703f0 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd659c700 1 --2- 172.21.15.86:0/2704300271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ccc00b5f0 tx=0x7f0ccc006ee0).ready entity=mon.0 client_cookie=6400a9823ef03466 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd559a700 1 -- 172.21.15.86:0/2704300271 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc00d600 con 0x7f0cd80703f0 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.445+0000 7f0cd559a700 1 -- 172.21.15.86:0/2704300271 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0ccc00dbd0 con 0x7f0cd80703f0 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cd559a700 1 -- 172.21.15.86:0/2704300271 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc011c00 con 0x7f0cd80703f0 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2704300271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 msgr2=0x7f0cd8100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:02.452 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2704300271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ccc00b5f0 tx=0x7f0ccc006ee0).stop 2023-11-15T05:01:02.453 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2704300271 shutdown_connections 2023-11-15T05:01:02.453 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2704300271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd8100d20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:02.453 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2704300271 >> 172.21.15.86:0/2704300271 conn(0x7f0cd80f63c0 msgr2=0x7f0cd80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:02.453 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2704300271 shutdown_connections 2023-11-15T05:01:02.453 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.446+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2704300271 wait complete. 2023-11-15T05:01:02.454 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cdc84b700 1 Processor -- start 2023-11-15T05:01:02.454 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cdc84b700 1 -- start start 2023-11-15T05:01:02.454 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cdc84b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:02.454 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cdc84b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd810d1e0 con 0x7f0cd80703f0 2023-11-15T05:01:02.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:02.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44682/0 (socket says 172.21.15.86:44682) 2023-11-15T05:01:02.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cd659c700 1 -- 172.21.15.86:0/2677892448 learned_addr learned my addr 172.21.15.86:0/2677892448 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:02.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cd659c700 1 -- 172.21.15.86:0/2677892448 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc00b200 con 0x7f0cd80703f0 2023-11-15T05:01:02.455 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cd659c700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0ccc00e040 tx=0x7f0ccc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc00d9f0 con 0x7f0cd80703f0 2023-11-15T05:01:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0ccc01a9f0 con 0x7f0cd80703f0 2023-11-15T05:01:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd810d3e0 con 0x7f0cd80703f0 2023-11-15T05:01:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.447+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ccc01b8c0 con 0x7f0cd80703f0 2023-11-15T05:01:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.448+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd810d7f0 con 0x7f0cd80703f0 2023-11-15T05:01:02.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.448+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8045ad0 con 0x7f0cd80703f0 2023-11-15T05:01:02.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.449+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0ccc00f070 con 0x7f0cd80703f0 2023-11-15T05:01:02.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.449+0000 7f0cc6ffd700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 0x7f0cbc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:02.457 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.449+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0ccc040d30 con 0x7f0cd80703f0 2023-11-15T05:01:02.458 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.449+0000 7f0cd5d9b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 0x7f0cbc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:02.460 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.449+0000 7f0cd5d9b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 0x7f0cbc032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:02.461 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.453+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ccc01ab60 con 0x7f0cd80703f0 2023-11-15T05:01:02.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:01.390403+0000 mgr.smithi086.hkfamo (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-11-15T05:01:02.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:01.753513+0000 mon.smithi086 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:02.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:01.755199+0000 mon.smithi086 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:02.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.607+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cd810e050 con 0x7f0cbc02fd50 2023-11-15T05:01:02.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.608+0000 7f0cc6ffd700 1 -- 172.21.15.86:0/2677892448 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0cd810e050 con 0x7f0cbc02fd50 2023-11-15T05:01:02.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrDij/hiR7eSH2Qx9862ufcWnXcZFHDbQK7V5eOuqjACN8uqNohFy8aCupjmHOvbnHOmvXU5a3xOuJCAF/YydfMuml7ijTtGia+x/QrrnY2ghivPGRSxCK0QX+PfeKSBhH3ogwoc2Rz4AITt7fbG7ZI0cnFxBQrtHsaelxsS+W68prLz2dfex1YBS3DvDlcMFWAMAwe970XcfyKQ5dsU/WJ7xvWkSlVlIGSrx6XiqPyxCdqZBPL2t34WHzGE3pCQZ8xNYSIpGQc+IBCkB3GB2dDX6Y9z/1OK1ezIimcd7TeSZHmurqPSWn49XQ0K4XG5/t4goPM5pcl3SfVOjldaepvUIJO+QP2sVnr3EDYLKBmKrJvASrJq11yk3v2xvR+Lki3m0u6a6dBN3yYrS883/dF7uEC8uhIZcpYpnDE6+e6iQXF8J5+zxgvUVOe1iRBvBspErgA3sXCEu8Qra8StOv/CpiX/P01nSzEvJsTqR/c8UMOToTz5L6y5Br3AeO/lk= ceph-d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:01:02.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.609+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 msgr2=0x7f0cbc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:02.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.609+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 0x7f0cbc032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0cc80096a0 tx=0x7f0cc8006b40).stop 2023-11-15T05:01:02.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.609+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 msgr2=0x7f0cd810ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:02.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.609+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0ccc00e040 tx=0x7f0ccc006ee0).stop 2023-11-15T05:01:02.611 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 shutdown_connections 2023-11-15T05:01:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f0cbc02fd50 0x7f0cbc032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 --2- 172.21.15.86:0/2677892448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cd80703f0 0x7f0cd810ccf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 >> 172.21.15.86:0/2677892448 conn(0x7f0cd80f63c0 msgr2=0x7f0cd80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:02.612 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 shutdown_connections 2023-11-15T05:01:02.613 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:02.610+0000 7f0cdc84b700 1 -- 172.21.15.86:0/2677892448 wait complete. 2023-11-15T05:01:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:01:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T05:01:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:Adding host smithi086... 2023-11-15T05:01:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi086 172.21.15.86 2023-11-15T05:01:03.288 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.284+0000 7fcea7556700 1 Processor -- start 2023-11-15T05:01:03.288 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.284+0000 7fcea7556700 1 -- start start 2023-11-15T05:01:03.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea7556700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:03.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea7556700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00fc5c0 con 0x7fcea0070390 2023-11-15T05:01:03.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea6554700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:03.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea6554700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44690/0 (socket says 172.21.15.86:44690) 2023-11-15T05:01:03.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea6554700 1 -- 172.21.15.86:0/3600985528 learned_addr learned my addr 172.21.15.86:0/3600985528 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:03.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea6554700 1 -- 172.21.15.86:0/3600985528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea01010c0 con 0x7fcea0070390 2023-11-15T05:01:03.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea6554700 1 --2- 172.21.15.86:0/3600985528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fce9c00b5f0 tx=0x7fce9c006ee0).ready entity=mon.0 client_cookie=d3dc48ed770206f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:03.290 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea5552700 1 -- 172.21.15.86:0/3600985528 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c00d600 con 0x7fcea0070390 2023-11-15T05:01:03.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea5552700 1 -- 172.21.15.86:0/3600985528 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce9c00dbd0 con 0x7fcea0070390 2023-11-15T05:01:03.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.285+0000 7fcea5552700 1 -- 172.21.15.86:0/3600985528 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c011c00 con 0x7fcea0070390 2023-11-15T05:01:03.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- 172.21.15.86:0/3600985528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 msgr2=0x7fcea0100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:03.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 --2- 172.21.15.86:0/3600985528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fce9c00b5f0 tx=0x7fce9c006ee0).stop 2023-11-15T05:01:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- 172.21.15.86:0/3600985528 shutdown_connections 2023-11-15T05:01:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 --2- 172.21.15.86:0/3600985528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea0100bd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- 172.21.15.86:0/3600985528 >> 172.21.15.86:0/3600985528 conn(0x7fcea00f6210 msgr2=0x7fcea00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- 172.21.15.86:0/3600985528 shutdown_connections 2023-11-15T05:01:03.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- 172.21.15.86:0/3600985528 wait complete. 2023-11-15T05:01:03.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 Processor -- start 2023-11-15T05:01:03.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.286+0000 7fcea7556700 1 -- start start 2023-11-15T05:01:03.293 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea7556700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:03.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea7556700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea010cf60 con 0x7fcea0070390 2023-11-15T05:01:03.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea6554700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:03.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea6554700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44702/0 (socket says 172.21.15.86:44702) 2023-11-15T05:01:03.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea6554700 1 -- 172.21.15.86:0/3372536985 learned_addr learned my addr 172.21.15.86:0/3372536985 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea6554700 1 -- 172.21.15.86:0/3372536985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c00b200 con 0x7fcea0070390 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea6554700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fce9c00e040 tx=0x7fce9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c00d9f0 con 0x7fcea0070390 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce9c01a9f0 con 0x7fcea0070390 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea010d160 con 0x7fcea0070390 2023-11-15T05:01:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9c01b8c0 con 0x7fcea0070390 2023-11-15T05:01:03.296 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.287+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea010d570 con 0x7fcea0070390 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.288+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0045ad0 con 0x7fcea0070390 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.288+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fce9c00f070 con 0x7fcea0070390 2023-11-15T05:01:03.297 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.289+0000 7fce92ffd700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 0x7fce88032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.289+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fce9c041a40 con 0x7fcea0070390 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.289+0000 7fcea5d53700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 0x7fce88032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:03.298 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.289+0000 7fcea5d53700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 0x7fce88032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fce940096a0 tx=0x7fce94006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:03.299 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.292+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce9c02d7a0 con 0x7fcea0070390 2023-11-15T05:01:03.462 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:03.462+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea010de50 con 0x7fce8802fdb0 2023-11-15T05:01:03.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:02.608628+0000 mgr.smithi086.hkfamo (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:03.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:03.463094+0000 mgr.smithi086.hkfamo (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:04.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Added host 'smithi086' 2023-11-15T05:01:04.050 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.048+0000 7fce92ffd700 1 -- 172.21.15.86:0/3372536985 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fcea010de50 con 0x7fce8802fdb0 2023-11-15T05:01:04.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 msgr2=0x7fce88032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:04.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 0x7fce88032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fce940096a0 tx=0x7fce94006b40).stop 2023-11-15T05:01:04.052 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 msgr2=0x7fcea010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:04.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fce9c00e040 tx=0x7fce9c006ee0).stop 2023-11-15T05:01:04.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 shutdown_connections 2023-11-15T05:01:04.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fce8802fdb0 0x7fce88032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:04.053 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 --2- 172.21.15.86:0/3372536985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcea0070390 0x7fcea010ca70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:04.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 >> 172.21.15.86:0/3372536985 conn(0x7fcea00f6210 msgr2=0x7fcea00f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:04.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 shutdown_connections 2023-11-15T05:01:04.054 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.051+0000 7fcea7556700 1 -- 172.21.15.86:0/3372536985 wait complete. 2023-11-15T05:01:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying mon service with default placement... 2023-11-15T05:01:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T05:01:04.716 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf6b39700 1 Processor -- start 2023-11-15T05:01:04.717 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf6b39700 1 -- start start 2023-11-15T05:01:04.717 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf6b39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:04.717 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf6b39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00fedd0 con 0x7f9cf0070430 2023-11-15T05:01:04.717 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf48d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:04.718 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf48d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44708/0 (socket says 172.21.15.86:44708) 2023-11-15T05:01:04.718 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf48d5700 1 -- 172.21.15.86:0/2929916663 learned_addr learned my addr 172.21.15.86:0/2929916663 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:04.718 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf48d5700 1 -- 172.21.15.86:0/2929916663 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00fef10 con 0x7f9cf0070430 2023-11-15T05:01:04.718 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.713+0000 7f9cf48d5700 1 --2- 172.21.15.86:0/2929916663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9ce400b5f0 tx=0x7f9ce4006ee0).ready entity=mon.0 client_cookie=3ec1d67754cbc18 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:04.719 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cef7fe700 1 -- 172.21.15.86:0/2929916663 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce400d600 con 0x7f9cf0070430 2023-11-15T05:01:04.719 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cef7fe700 1 -- 172.21.15.86:0/2929916663 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9ce400dbd0 con 0x7f9cf0070430 2023-11-15T05:01:04.719 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cef7fe700 1 -- 172.21.15.86:0/2929916663 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce4011c00 con 0x7f9cf0070430 2023-11-15T05:01:04.719 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 -- 172.21.15.86:0/2929916663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 msgr2=0x7f9cf00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:04.720 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/2929916663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9ce400b5f0 tx=0x7f9ce4006ee0).stop 2023-11-15T05:01:04.720 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 -- 172.21.15.86:0/2929916663 shutdown_connections 2023-11-15T05:01:04.720 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/2929916663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf00fe8e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:04.720 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 -- 172.21.15.86:0/2929916663 >> 172.21.15.86:0/2929916663 conn(0x7f9cf00f63c0 msgr2=0x7f9cf00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:04.721 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 -- 172.21.15.86:0/2929916663 shutdown_connections 2023-11-15T05:01:04.721 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.714+0000 7f9cf6b39700 1 -- 172.21.15.86:0/2929916663 wait complete. 2023-11-15T05:01:04.721 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf6b39700 1 Processor -- start 2023-11-15T05:01:04.721 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf6b39700 1 -- start start 2023-11-15T05:01:04.721 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf6b39700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:04.722 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf6b39700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf010af30 con 0x7f9cf0070430 2023-11-15T05:01:04.722 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf48d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:04.722 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf48d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44720/0 (socket says 172.21.15.86:44720) 2023-11-15T05:01:04.722 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.715+0000 7f9cf48d5700 1 -- 172.21.15.86:0/3472652131 learned_addr learned my addr 172.21.15.86:0/3472652131 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:04.722 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9cf48d5700 1 -- 172.21.15.86:0/3472652131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce400b200 con 0x7f9cf0070430 2023-11-15T05:01:04.723 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9cf48d5700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ce400e040 tx=0x7f9ce4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:04.723 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce400d9f0 con 0x7f9cf0070430 2023-11-15T05:01:04.723 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9ce401a9f0 con 0x7f9cf0070430 2023-11-15T05:01:04.723 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf010b130 con 0x7f9cf0070430 2023-11-15T05:01:04.723 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce401b8c0 con 0x7f9cf0070430 2023-11-15T05:01:04.724 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.716+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf010b540 con 0x7f9cf0070430 2023-11-15T05:01:04.725 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.717+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9ce400f070 con 0x7f9cf0070430 2023-11-15T05:01:04.725 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.717+0000 7f9ced7fa700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 0x7f9cd4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:04.725 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.717+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9ce40412e0 con 0x7f9cf0070430 2023-11-15T05:01:04.725 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.717+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0045ad0 con 0x7f9cf0070430 2023-11-15T05:01:04.726 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.717+0000 7f9ceffff700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 0x7f9cd4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:04.726 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.718+0000 7f9ceffff700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 0x7f9cd4032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9cdc0096a0 tx=0x7f9cdc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:04.726 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.721+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ce40224f0 con 0x7f9cf0070430 2023-11-15T05:01:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.895+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cf0104900 con 0x7f9cd402fd50 2023-11-15T05:01:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.899+0000 7f9ced7fa700 1 -- 172.21.15.86:0/3472652131 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9cf0104900 con 0x7f9cd402fd50 2023-11-15T05:01:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T05:01:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 msgr2=0x7f9cd4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 0x7f9cd4032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9cdc0096a0 tx=0x7f9cdc006b40).stop 2023-11-15T05:01:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 msgr2=0x7f9cf010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ce400e040 tx=0x7f9ce4006ee0).stop 2023-11-15T05:01:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 shutdown_connections 2023-11-15T05:01:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f9cd402fd50 0x7f9cd4032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 --2- 172.21.15.86:0/3472652131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf0070430 0x7f9cf010aa40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 >> 172.21.15.86:0/3472652131 conn(0x7f9cf00f63c0 msgr2=0x7f9cf00f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 shutdown_connections 2023-11-15T05:01:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:04.901+0000 7f9cf6b39700 1 -- 172.21.15.86:0/3472652131 wait complete. 2023-11-15T05:01:05.183 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:04.048163+0000 mon.smithi086 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:05.183 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:04.048459+0000 mgr.smithi086.hkfamo (mgr.14114) 9 : cephadm [INF] Added host smithi086 2023-11-15T05:01:05.183 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:04.048680+0000 mon.smithi086 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:01:05.184 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:04.899461+0000 mon.smithi086 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:05.248 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying mgr service with default placement... 2023-11-15T05:01:05.248 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T05:01:05.595 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.593+0000 7f61b5696700 1 Processor -- start 2023-11-15T05:01:05.595 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.593+0000 7f61b5696700 1 -- start start 2023-11-15T05:01:05.595 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61b5696700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:05.596 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61b5696700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00728b0 con 0x7f61b0072490 2023-11-15T05:01:05.596 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:05.597 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44736/0 (socket says 172.21.15.86:44736) 2023-11-15T05:01:05.597 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61aeffd700 1 -- 172.21.15.86:0/1718174328 learned_addr learned my addr 172.21.15.86:0/1718174328 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:05.597 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61aeffd700 1 -- 172.21.15.86:0/1718174328 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0070f20 con 0x7f61b0072490 2023-11-15T05:01:05.598 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61aeffd700 1 --2- 172.21.15.86:0/1718174328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f61a400b5f0 tx=0x7f61a4006ee0).ready entity=mon.0 client_cookie=ce575f765b1e9fe6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:05.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61adffb700 1 -- 172.21.15.86:0/1718174328 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a400d600 con 0x7f61b0072490 2023-11-15T05:01:05.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61adffb700 1 -- 172.21.15.86:0/1718174328 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f61a400dbd0 con 0x7f61b0072490 2023-11-15T05:01:05.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.594+0000 7f61adffb700 1 -- 172.21.15.86:0/1718174328 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a4011c00 con 0x7f61b0072490 2023-11-15T05:01:05.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 -- 172.21.15.86:0/1718174328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 msgr2=0x7f61b0070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:05.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 --2- 172.21.15.86:0/1718174328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f61a400b5f0 tx=0x7f61a4006ee0).stop 2023-11-15T05:01:05.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 -- 172.21.15.86:0/1718174328 shutdown_connections 2023-11-15T05:01:05.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 --2- 172.21.15.86:0/1718174328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b0070a30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:05.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 -- 172.21.15.86:0/1718174328 >> 172.21.15.86:0/1718174328 conn(0x7f61b00f74c0 msgr2=0x7f61b00f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:05.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 -- 172.21.15.86:0/1718174328 shutdown_connections 2023-11-15T05:01:05.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.595+0000 7f61b5696700 1 -- 172.21.15.86:0/1718174328 wait complete. 2023-11-15T05:01:05.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.596+0000 7f61b5696700 1 Processor -- start 2023-11-15T05:01:05.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.596+0000 7f61b5696700 1 -- start start 2023-11-15T05:01:05.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.596+0000 7f61b5696700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:05.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.596+0000 7f61b5696700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b010d1f0 con 0x7f61b0072490 2023-11-15T05:01:05.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.596+0000 7f61aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:05.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44750/0 (socket says 172.21.15.86:44750) 2023-11-15T05:01:05.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61aeffd700 1 -- 172.21.15.86:0/3771525398 learned_addr learned my addr 172.21.15.86:0/3771525398 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:05.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61aeffd700 1 -- 172.21.15.86:0/3771525398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a400b200 con 0x7f61b0072490 2023-11-15T05:01:05.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61aeffd700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f61a4011820 tx=0x7f61a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:05.606 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a401b410 con 0x7f61b0072490 2023-11-15T05:01:05.606 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f61a401b9e0 con 0x7f61b0072490 2023-11-15T05:01:05.606 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b010d3f0 con 0x7f61b0072490 2023-11-15T05:01:05.606 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a401a8b0 con 0x7f61b0072490 2023-11-15T05:01:05.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.597+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b010d800 con 0x7f61b0072490 2023-11-15T05:01:05.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f61a400f070 con 0x7f61b0072490 2023-11-15T05:01:05.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61977fe700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 0x7f619c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:05.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f61a4042160 con 0x7f61b0072490 2023-11-15T05:01:05.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b0045ad0 con 0x7f61b0072490 2023-11-15T05:01:05.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61ae7fc700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 0x7f619c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:05.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.598+0000 7f61ae7fc700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 0x7f619c032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f61a00096a0 tx=0x7f61a0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:05.610 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.602+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61a4016070 con 0x7f61b0072490 2023-11-15T05:01:05.777 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.777+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b010ed30 con 0x7f619c02fd50 2023-11-15T05:01:05.781 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.781+0000 7f61977fe700 1 -- 172.21.15.86:0/3771525398 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f61b010ed30 con 0x7f619c02fd50 2023-11-15T05:01:05.781 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T05:01:05.783 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 msgr2=0x7f619c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:05.783 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 0x7f619c032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f61a00096a0 tx=0x7f61a0006b40).stop 2023-11-15T05:01:05.784 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 msgr2=0x7f61b010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:05.784 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f61a4011820 tx=0x7f61a4006ee0).stop 2023-11-15T05:01:05.784 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 shutdown_connections 2023-11-15T05:01:05.784 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f619c02fd50 0x7f619c032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:05.784 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 --2- 172.21.15.86:0/3771525398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61b0072490 0x7f61b010cd00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:05.785 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 >> 172.21.15.86:0/3771525398 conn(0x7f61b00f74c0 msgr2=0x7f61b00f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:05.785 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 shutdown_connections 2023-11-15T05:01:05.785 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:05.782+0000 7f61b5696700 1 -- 172.21.15.86:0/3771525398 wait complete. 2023-11-15T05:01:06.083 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:04.896477+0000 mgr.smithi086.hkfamo (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:06.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:04.897197+0000 mgr.smithi086.hkfamo (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T05:01:06.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:05.781230+0000 mon.smithi086 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:06.115 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying crash service with default placement... 2023-11-15T05:01:06.115 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T05:01:06.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.467+0000 7fa813503700 1 Processor -- start 2023-11-15T05:01:06.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.467+0000 7fa813503700 1 -- start start 2023-11-15T05:01:06.474 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.467+0000 7fa813503700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:06.475 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa813503700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c0fc730 con 0x7fa80c0703d0 2023-11-15T05:01:06.475 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa81129f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:06.475 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa81129f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44760/0 (socket says 172.21.15.86:44760) 2023-11-15T05:01:06.475 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa81129f700 1 -- 172.21.15.86:0/1108209784 learned_addr learned my addr 172.21.15.86:0/1108209784 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa81129f700 1 -- 172.21.15.86:0/1108209784 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c101230 con 0x7fa80c0703d0 2023-11-15T05:01:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa81129f700 1 --2- 172.21.15.86:0/1108209784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa7fc00b5f0 tx=0x7fa7fc006ee0).ready entity=mon.0 client_cookie=cb706198d070a0bf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa7fbfff700 1 -- 172.21.15.86:0/1108209784 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc00d600 con 0x7fa80c0703d0 2023-11-15T05:01:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa7fbfff700 1 -- 172.21.15.86:0/1108209784 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa7fc00dbd0 con 0x7fa80c0703d0 2023-11-15T05:01:06.477 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.468+0000 7fa7fbfff700 1 -- 172.21.15.86:0/1108209784 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc011c00 con 0x7fa80c0703d0 2023-11-15T05:01:06.477 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 -- 172.21.15.86:0/1108209784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 msgr2=0x7fa80c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:06.477 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 --2- 172.21.15.86:0/1108209784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fa7fc00b5f0 tx=0x7fa7fc006ee0).stop 2023-11-15T05:01:06.477 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 -- 172.21.15.86:0/1108209784 shutdown_connections 2023-11-15T05:01:06.478 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 --2- 172.21.15.86:0/1108209784 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c100d40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:06.478 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 -- 172.21.15.86:0/1108209784 >> 172.21.15.86:0/1108209784 conn(0x7fa80c0f63c0 msgr2=0x7fa80c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:06.478 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 -- 172.21.15.86:0/1108209784 shutdown_connections 2023-11-15T05:01:06.479 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.469+0000 7fa813503700 1 -- 172.21.15.86:0/1108209784 wait complete. 2023-11-15T05:01:06.479 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa813503700 1 Processor -- start 2023-11-15T05:01:06.479 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa813503700 1 -- start start 2023-11-15T05:01:06.479 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa813503700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:06.480 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa813503700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c10d0a0 con 0x7fa80c0703d0 2023-11-15T05:01:06.480 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa81129f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:06.480 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa81129f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44768/0 (socket says 172.21.15.86:44768) 2023-11-15T05:01:06.480 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa81129f700 1 -- 172.21.15.86:0/3732746398 learned_addr learned my addr 172.21.15.86:0/3732746398 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:06.480 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa81129f700 1 -- 172.21.15.86:0/3732746398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc00b200 con 0x7fa80c0703d0 2023-11-15T05:01:06.481 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.470+0000 7fa81129f700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa7fc00b190 tx=0x7fa7fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:06.481 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc01b410 con 0x7fa80c0703d0 2023-11-15T05:01:06.481 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c10d2a0 con 0x7fa80c0703d0 2023-11-15T05:01:06.482 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa7fc01b9e0 con 0x7fa80c0703d0 2023-11-15T05:01:06.482 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7fc01a880 con 0x7fa80c0703d0 2023-11-15T05:01:06.482 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c10d6b0 con 0x7fa80c0703d0 2023-11-15T05:01:06.482 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.471+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa7fc00f070 con 0x7fa80c0703d0 2023-11-15T05:01:06.483 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.472+0000 7fa7f9ffb700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 0x7fa7f4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:06.484 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.472+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa7fc041360 con 0x7fa80c0703d0 2023-11-15T05:01:06.484 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.472+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c045ad0 con 0x7fa80c0703d0 2023-11-15T05:01:06.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.472+0000 7fa810a9e700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 0x7fa7f4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:06.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.472+0000 7fa810a9e700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 0x7fa7f4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa8080096a0 tx=0x7fa808006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:06.486 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.476+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7fc016070 con 0x7fa80c0703d0 2023-11-15T05:01:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.651+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa80c106b00 con 0x7fa7f402fd50 2023-11-15T05:01:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.657+0000 7fa7f9ffb700 1 -- 172.21.15.86:0/3732746398 <== mgr.14114 v2:172.21.15.86:6800/2368880557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa80c106b00 con 0x7fa7f402fd50 2023-11-15T05:01:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T05:01:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 msgr2=0x7fa7f4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 0x7fa7f4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa8080096a0 tx=0x7fa808006b40).stop 2023-11-15T05:01:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 msgr2=0x7fa80c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa7fc00b190 tx=0x7fa7fc006ee0).stop 2023-11-15T05:01:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 shutdown_connections 2023-11-15T05:01:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7fa7f402fd50 0x7fa7f4032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 --2- 172.21.15.86:0/3732746398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa80c0703d0 0x7fa80c10cbb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.659+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 >> 172.21.15.86:0/3732746398 conn(0x7fa80c0f63c0 msgr2=0x7fa80c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.660+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 shutdown_connections 2023-11-15T05:01:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:06.660+0000 7fa813503700 1 -- 172.21.15.86:0/3732746398 wait complete. 2023-11-15T05:01:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:Enabling mgr prometheus module... 2023-11-15T05:01:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T05:01:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.365+0000 7f66d60d2700 1 Processor -- start 2023-11-15T05:01:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.365+0000 7f66d60d2700 1 -- start start 2023-11-15T05:01:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.365+0000 7f66d60d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.365+0000 7f66d60d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00fee10 con 0x7f66d00703f0 2023-11-15T05:01:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.365+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44774/0 (socket says 172.21.15.86:44774) 2023-11-15T05:01:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66cf7fe700 1 -- 172.21.15.86:0/4156956194 learned_addr learned my addr 172.21.15.86:0/4156956194 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66cf7fe700 1 -- 172.21.15.86:0/4156956194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00fef50 con 0x7f66d00703f0 2023-11-15T05:01:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66cf7fe700 1 --2- 172.21.15.86:0/4156956194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f66c400b5f0 tx=0x7f66c4006ee0).ready entity=mon.0 client_cookie=49570c9e3e9fd17f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66ce7fc700 1 -- 172.21.15.86:0/4156956194 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c400d600 con 0x7f66d00703f0 2023-11-15T05:01:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66ce7fc700 1 -- 172.21.15.86:0/4156956194 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f66c400dbd0 con 0x7f66d00703f0 2023-11-15T05:01:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66ce7fc700 1 -- 172.21.15.86:0/4156956194 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c4011c00 con 0x7f66d00703f0 2023-11-15T05:01:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.366+0000 7f66d60d2700 1 -- 172.21.15.86:0/4156956194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 msgr2=0x7f66d00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 --2- 172.21.15.86:0/4156956194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f66c400b5f0 tx=0x7f66c4006ee0).stop 2023-11-15T05:01:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 -- 172.21.15.86:0/4156956194 shutdown_connections 2023-11-15T05:01:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 --2- 172.21.15.86:0/4156956194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d00fe920 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 -- 172.21.15.86:0/4156956194 >> 172.21.15.86:0/4156956194 conn(0x7f66d00f63c0 msgr2=0x7f66d00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 -- 172.21.15.86:0/4156956194 shutdown_connections 2023-11-15T05:01:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 -- 172.21.15.86:0/4156956194 wait complete. 2023-11-15T05:01:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 Processor -- start 2023-11-15T05:01:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 -- start start 2023-11-15T05:01:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.367+0000 7f66d60d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66d60d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d010aeb0 con 0x7f66d00703f0 2023-11-15T05:01:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:07.376 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44786/0 (socket says 172.21.15.86:44786) 2023-11-15T05:01:07.376 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66cf7fe700 1 -- 172.21.15.86:0/2685954844 learned_addr learned my addr 172.21.15.86:0/2685954844 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:07.376 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66cf7fe700 1 -- 172.21.15.86:0/2685954844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c400b200 con 0x7f66d00703f0 2023-11-15T05:01:07.376 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66cf7fe700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f66c400b190 tx=0x7f66c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:07.376 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c401b410 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d010b0b0 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f66c401b9e0 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c401a880 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.368+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d010b4c0 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.369+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f66c400f070 con 0x7f66d00703f0 2023-11-15T05:01:07.377 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.369+0000 7f66b7fff700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 0x7f66bc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:07.378 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.369+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0045ad0 con 0x7f66d00703f0 2023-11-15T05:01:07.378 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.369+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f66c40411d0 con 0x7f66d00703f0 2023-11-15T05:01:07.378 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.369+0000 7f66ceffd700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 0x7f66bc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:07.378 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.370+0000 7f66ceffd700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 0x7f66bc032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f66c00096a0 tx=0x7f66c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:07.379 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.373+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66c4016070 con 0x7f66d00703f0 2023-11-15T05:01:07.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:05.778518+0000 mgr.smithi086.hkfamo (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:07.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:05.779022+0000 mgr.smithi086.hkfamo (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T05:01:07.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:06.657620+0000 mon.smithi086 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.86:0/3913024690' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:07.562 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:07.561+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f66d00ffbd0 con 0x7f66d00703f0 2023-11-15T05:01:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.266+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f66c4025460 con 0x7f66d00703f0 2023-11-15T05:01:08.267 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.267+0000 7f66b7fff700 1 -- 172.21.15.86:0/2685954844 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f66c4024b70 con 0x7f66d00703f0 2023-11-15T05:01:08.270 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 msgr2=0x7f66bc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.270 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 0x7f66bc032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f66c00096a0 tx=0x7f66c0006b40).stop 2023-11-15T05:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 msgr2=0x7f66d010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f66c400b190 tx=0x7f66c4006ee0).stop 2023-11-15T05:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 shutdown_connections 2023-11-15T05:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7f66bc02fdb0 0x7f66bc032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 --2- 172.21.15.86:0/2685954844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d00703f0 0x7f66d010a9c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.272 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 >> 172.21.15.86:0/2685954844 conn(0x7f66d00f63c0 msgr2=0x7f66d00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:08.272 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 shutdown_connections 2023-11-15T05:01:08.272 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.269+0000 7f66d60d2700 1 -- 172.21.15.86:0/2685954844 wait complete. 2023-11-15T05:01:08.559 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:06.652859+0000 mgr.smithi086.hkfamo (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:08.559 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:06.653532+0000 mgr.smithi086.hkfamo (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T05:01:08.559 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:07.561582+0000 mon.smithi086 (mon.0) 58 : audit [INF] from='client.? 172.21.15.86:0/2685954844' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T05:01:08.614 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying prometheus service with default placement... 2023-11-15T05:01:08.614 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T05:01:08.980 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.979+0000 7efeb8de1700 1 Processor -- start 2023-11-15T05:01:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.979+0000 7efeb8de1700 1 -- start start 2023-11-15T05:01:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.979+0000 7efeb8de1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.979+0000 7efeb8de1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40fb490 con 0x7efeb40fab80 2023-11-15T05:01:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.979+0000 7efeb259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46360/0 (socket says 172.21.15.86:46360) 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb259c700 1 -- 172.21.15.86:0/3815729804 learned_addr learned my addr 172.21.15.86:0/3815729804 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb259c700 1 -- 172.21.15.86:0/3815729804 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40fb5d0 con 0x7efeb40fab80 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb259c700 1 --2- 172.21.15.86:0/3815729804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7efea400b5f0 tx=0x7efea4006ee0).ready entity=mon.0 client_cookie=f4be63bb2c15523c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb159a700 1 -- 172.21.15.86:0/3815729804 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea400d600 con 0x7efeb40fab80 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb159a700 1 -- 172.21.15.86:0/3815729804 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efea400dbd0 con 0x7efeb40fab80 2023-11-15T05:01:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.980+0000 7efeb159a700 1 -- 172.21.15.86:0/3815729804 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea4011c00 con 0x7efeb40fab80 2023-11-15T05:01:08.985 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- 172.21.15.86:0/3815729804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 msgr2=0x7efeb40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:08.985 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3815729804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7efea400b5f0 tx=0x7efea4006ee0).stop 2023-11-15T05:01:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- 172.21.15.86:0/3815729804 shutdown_connections 2023-11-15T05:01:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3815729804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb40fafa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- 172.21.15.86:0/3815729804 >> 172.21.15.86:0/3815729804 conn(0x7efeb40f63c0 msgr2=0x7efeb40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- 172.21.15.86:0/3815729804 shutdown_connections 2023-11-15T05:01:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- 172.21.15.86:0/3815729804 wait complete. 2023-11-15T05:01:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 Processor -- start 2023-11-15T05:01:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.981+0000 7efeb8de1700 1 -- start start 2023-11-15T05:01:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb8de1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb8de1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb410aef0 con 0x7efeb40fab80 2023-11-15T05:01:08.987 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46364/0 (socket says 172.21.15.86:46364) 2023-11-15T05:01:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb259c700 1 -- 172.21.15.86:0/3112563539 learned_addr learned my addr 172.21.15.86:0/3112563539 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb259c700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea400b200 con 0x7efeb40fab80 2023-11-15T05:01:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb259c700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7efea400b1d0 tx=0x7efea4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:08.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea401b410 con 0x7efeb40fab80 2023-11-15T05:01:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efea401b9e0 con 0x7efeb40fab80 2023-11-15T05:01:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb410b0f0 con 0x7efeb40fab80 2023-11-15T05:01:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efea401a840 con 0x7efeb40fab80 2023-11-15T05:01:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.982+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb410b500 con 0x7efeb40fab80 2023-11-15T05:01:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.983+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7efea400f070 con 0x7efeb40fab80 2023-11-15T05:01:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.983+0000 7efea2ffd700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.983+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efea4041240 con 0x7efeb40fab80 2023-11-15T05:01:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.983+0000 7efeb1d9b700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2368880557 2023-11-15T05:01:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.983+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4045ad0 con 0x7efeb40fab80 2023-11-15T05:01:08.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.984+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:01:08.993 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:08.988+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efea4025460 con 0x7efeb40fab80 2023-11-15T05:01:09.175 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:09.171+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efeb41049a0 con 0x7efe9802fe00 2023-11-15T05:01:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:09.185+0000 7efeb1d9b700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2368880557 2023-11-15T05:01:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:09.185+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:01:09.586 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:09.586+0000 7efeb1d9b700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2368880557 2023-11-15T05:01:09.586 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:09.586+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:01:09.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:08.267638+0000 mon.smithi086 (mon.0) 59 : audit [INF] from='client.? 172.21.15.86:0/2685954844' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T05:01:09.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:08.267696+0000 mon.smithi086 (mon.0) 60 : cluster [DBG] mgrmap e8: smithi086.hkfamo(active, since 17s) 2023-11-15T05:01:10.387 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:10.387+0000 7efeb1d9b700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2368880557 2023-11-15T05:01:10.387 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:10.387+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:01:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:11.988+0000 7efeb1d9b700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/2368880557 2023-11-15T05:01:11.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:11.988+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T05:01:13.274 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:13.273+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7efea40257e0 con 0x7efeb40fab80 2023-11-15T05:01:13.275 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:13.274+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 msgr2=0x7efe980322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:01:13.275 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:13.274+0000 7efea2ffd700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:13.222311+0000 mon.smithi086 (mon.0) 61 : cluster [INF] Active manager daemon smithi086.hkfamo restarted 2023-11-15T05:01:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:13.222684+0000 mon.smithi086 (mon.0) 62 : cluster [INF] Activating manager daemon smithi086.hkfamo 2023-11-15T05:01:13.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:13.224417+0000 mon.smithi086 (mon.0) 63 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T05:01:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.283+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7efea4043af0 con 0x7efeb40fab80 2023-11-15T05:01:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.283+0000 7efea2ffd700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 0x7efe98036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.283+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 --> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efeb41049a0 con 0x7efe98033c20 2023-11-15T05:01:14.286 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.284+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 0x7efe98036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.286 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.284+0000 7efeb1d9b700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 0x7efe98036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efea8006fd0 tx=0x7efea8006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.289+0000 7efea2ffd700 1 -- 172.21.15.86:0/3112563539 <== mgr.14142 v2:172.21.15.86:6800/1131374657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7efeb41049a0 con 0x7efe98033c20 2023-11-15T05:01:14.289 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T05:01:14.291 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 msgr2=0x7efe98036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 0x7efe98036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efea8006fd0 tx=0x7efea8006d30).stop 2023-11-15T05:01:14.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 msgr2=0x7efeb410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.292 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7efea400b1d0 tx=0x7efea4006ee0).stop 2023-11-15T05:01:14.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 shutdown_connections 2023-11-15T05:01:14.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7efe98033c20 0x7efe98036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efeb40fab80 0x7efeb410aa00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.294 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 --2- 172.21.15.86:0/3112563539 >> [v2:172.21.15.86:6800/2368880557,v1:172.21.15.86:6801/2368880557] conn(0x7efe9802fe00 0x7efe980322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:14.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 >> 172.21.15.86:0/3112563539 conn(0x7efeb40f63c0 msgr2=0x7efeb40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:14.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 shutdown_connections 2023-11-15T05:01:14.295 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.291+0000 7efeb8de1700 1 -- 172.21.15.86:0/3112563539 wait complete. 2023-11-15T05:01:14.575 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:13.275074+0000 mon.smithi086 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi086.hkfamo(active, starting, since 0.0524685s) 2023-11-15T05:01:14.575 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.276205+0000 mon.smithi086 (mon.0) 65 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2023-11-15T05:01:14.576 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.276779+0000 mon.smithi086 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mgr metadata", "who": "smithi086.hkfamo", "id": "smithi086.hkfamo"}]: dispatch 2023-11-15T05:01:14.576 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.278926+0000 mon.smithi086 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:01:14.576 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.279117+0000 mon.smithi086 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:01:14.576 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.279263+0000 mon.smithi086 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:01:14.576 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:13.292684+0000 mon.smithi086 (mon.0) 70 : cluster [INF] Manager daemon smithi086.hkfamo is now available 2023-11-15T05:01:14.577 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.387987+0000 mon.smithi086 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:01:14.577 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.389696+0000 mon.smithi086 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:01:14.577 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.403691+0000 mon.smithi086 (mon.0) 73 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:01:14.577 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:13.406367+0000 mon.smithi086 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:01:14.584 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying grafana service with default placement... 2023-11-15T05:01:14.584 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T05:01:14.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.978+0000 7f54bfa76700 1 Processor -- start 2023-11-15T05:01:14.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bfa76700 1 -- start start 2023-11-15T05:01:14.981 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bfa76700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bfa76700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b80fc770 con 0x7f54b8070390 2023-11-15T05:01:14.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bd812700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bd812700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46408/0 (socket says 172.21.15.86:46408) 2023-11-15T05:01:14.982 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bd812700 1 -- 172.21.15.86:0/929678288 learned_addr learned my addr 172.21.15.86:0/929678288 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:14.983 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bd812700 1 -- 172.21.15.86:0/929678288 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8101270 con 0x7f54b8070390 2023-11-15T05:01:14.983 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.979+0000 7f54bd812700 1 --2- 172.21.15.86:0/929678288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54a800b5f0 tx=0x7f54a8006ee0).ready entity=mon.0 client_cookie=51f2ecc6ff67385 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.983 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bc810700 1 -- 172.21.15.86:0/929678288 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54a800d600 con 0x7f54b8070390 2023-11-15T05:01:14.983 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bc810700 1 -- 172.21.15.86:0/929678288 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f54a800dbd0 con 0x7f54b8070390 2023-11-15T05:01:14.983 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bc810700 1 -- 172.21.15.86:0/929678288 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54a8011c00 con 0x7f54b8070390 2023-11-15T05:01:14.984 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 -- 172.21.15.86:0/929678288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 msgr2=0x7f54b8100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:14.984 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 --2- 172.21.15.86:0/929678288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54a800b5f0 tx=0x7f54a8006ee0).stop 2023-11-15T05:01:14.984 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 -- 172.21.15.86:0/929678288 shutdown_connections 2023-11-15T05:01:14.984 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 --2- 172.21.15.86:0/929678288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b8100d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:14.984 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 -- 172.21.15.86:0/929678288 >> 172.21.15.86:0/929678288 conn(0x7f54b80f63c0 msgr2=0x7f54b80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:14.985 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 -- 172.21.15.86:0/929678288 shutdown_connections 2023-11-15T05:01:14.985 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.980+0000 7f54bfa76700 1 -- 172.21.15.86:0/929678288 wait complete. 2023-11-15T05:01:14.985 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bfa76700 1 Processor -- start 2023-11-15T05:01:14.986 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bfa76700 1 -- start start 2023-11-15T05:01:14.988 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bfa76700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.988 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bfa76700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b810d200 con 0x7f54b8070390 2023-11-15T05:01:14.988 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bd812700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bd812700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46418/0 (socket says 172.21.15.86:46418) 2023-11-15T05:01:14.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bd812700 1 -- 172.21.15.86:0/57289335 learned_addr learned my addr 172.21.15.86:0/57289335 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:14.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.981+0000 7f54bd812700 1 -- 172.21.15.86:0/57289335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a800b200 con 0x7f54b8070390 2023-11-15T05:01:14.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54bd812700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f54a800e040 tx=0x7f54a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.989 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54a800d9f0 con 0x7f54b8070390 2023-11-15T05:01:14.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f54a801a9f0 con 0x7f54b8070390 2023-11-15T05:01:14.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b810d400 con 0x7f54b8070390 2023-11-15T05:01:14.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54a801b8c0 con 0x7f54b8070390 2023-11-15T05:01:14.990 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b810d810 con 0x7f54b8070390 2023-11-15T05:01:14.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.982+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f54a800f070 con 0x7f54b8070390 2023-11-15T05:01:14.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.983+0000 7f54a67fc700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 0x7f54a0032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:14.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.983+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f54a8041400 con 0x7f54b8070390 2023-11-15T05:01:14.991 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.983+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b8045ad0 con 0x7f54b8070390 2023-11-15T05:01:14.992 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.983+0000 7f54bd011700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 0x7f54a0032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:14.992 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.983+0000 7f54bd011700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 0x7f54a0032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54b40096a0 tx=0x7f54b4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:14.992 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:14.987+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54a8016070 con 0x7f54b8070390 2023-11-15T05:01:15.161 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.161+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 --> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f54b810ed00 con 0x7f54a002fd70 2023-11-15T05:01:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.165+0000 7f54a67fc700 1 -- 172.21.15.86:0/57289335 <== mgr.14142 v2:172.21.15.86:6800/1131374657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f54b810ed00 con 0x7f54a002fd70 2023-11-15T05:01:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T05:01:15.167 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 msgr2=0x7f54a0032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 0x7f54a0032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54b40096a0 tx=0x7f54b4006b40).stop 2023-11-15T05:01:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 msgr2=0x7f54b810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f54a800e040 tx=0x7f54a8006ee0).stop 2023-11-15T05:01:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 shutdown_connections 2023-11-15T05:01:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f54a002fd70 0x7f54a0032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 --2- 172.21.15.86:0/57289335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54b8070390 0x7f54b810cd10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 >> 172.21.15.86:0/57289335 conn(0x7f54b80f63c0 msgr2=0x7f54b80f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 shutdown_connections 2023-11-15T05:01:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.167+0000 7f54bfa76700 1 -- 172.21.15.86:0/57289335 wait complete. 2023-11-15T05:01:15.449 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:14.286050+0000 mon.smithi086 (mon.0) 75 : cluster [DBG] mgrmap e10: smithi086.hkfamo(active, since 1.06344s) 2023-11-15T05:01:15.449 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:14.286972+0000 mgr.smithi086.hkfamo (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T05:01:15.450 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:14.289029+0000 mon.smithi086 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:15.450 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:15.165179+0000 mon.smithi086 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:15.513 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying node-exporter service with default placement... 2023-11-15T05:01:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T05:01:15.853 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.847+0000 7f6d2da06700 1 Processor -- start 2023-11-15T05:01:15.853 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d2da06700 1 -- start start 2023-11-15T05:01:15.853 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d2da06700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:15.853 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d2da06700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d28072920 con 0x7f6d28072500 2023-11-15T05:01:15.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:15.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46434/0 (socket says 172.21.15.86:46434) 2023-11-15T05:01:15.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d26ffd700 1 -- 172.21.15.86:0/543721346 learned_addr learned my addr 172.21.15.86:0/543721346 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:15.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d26ffd700 1 -- 172.21.15.86:0/543721346 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28070f90 con 0x7f6d28072500 2023-11-15T05:01:15.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d26ffd700 1 --2- 172.21.15.86:0/543721346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6d1000b5f0 tx=0x7f6d10006ee0).ready entity=mon.0 client_cookie=d46a11ea2dd75603 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:15.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d25ffb700 1 -- 172.21.15.86:0/543721346 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d1000d600 con 0x7f6d28072500 2023-11-15T05:01:15.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d25ffb700 1 -- 172.21.15.86:0/543721346 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6d1000dbd0 con 0x7f6d28072500 2023-11-15T05:01:15.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.848+0000 7f6d25ffb700 1 -- 172.21.15.86:0/543721346 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d10011c00 con 0x7f6d28072500 2023-11-15T05:01:15.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 -- 172.21.15.86:0/543721346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 msgr2=0x7f6d28070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:15.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 --2- 172.21.15.86:0/543721346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6d1000b5f0 tx=0x7f6d10006ee0).stop 2023-11-15T05:01:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 -- 172.21.15.86:0/543721346 shutdown_connections 2023-11-15T05:01:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 --2- 172.21.15.86:0/543721346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d28070aa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 -- 172.21.15.86:0/543721346 >> 172.21.15.86:0/543721346 conn(0x7f6d280f74b0 msgr2=0x7f6d280f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 -- 172.21.15.86:0/543721346 shutdown_connections 2023-11-15T05:01:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.849+0000 7f6d2da06700 1 -- 172.21.15.86:0/543721346 wait complete. 2023-11-15T05:01:15.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d2da06700 1 Processor -- start 2023-11-15T05:01:15.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d2da06700 1 -- start start 2023-11-15T05:01:15.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d2da06700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:15.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d2da06700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2810d1f0 con 0x7f6d28072500 2023-11-15T05:01:15.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:15.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46444/0 (socket says 172.21.15.86:46444) 2023-11-15T05:01:15.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d26ffd700 1 -- 172.21.15.86:0/2684725289 learned_addr learned my addr 172.21.15.86:0/2684725289 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:15.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d26ffd700 1 -- 172.21.15.86:0/2684725289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1000b200 con 0x7f6d28072500 2023-11-15T05:01:15.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.850+0000 7f6d26ffd700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6d10011820 tx=0x7f6d10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:15.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.851+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d1001b410 con 0x7f6d28072500 2023-11-15T05:01:15.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.851+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6d1001b9e0 con 0x7f6d28072500 2023-11-15T05:01:15.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.851+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d1001a8b0 con 0x7f6d28072500 2023-11-15T05:01:15.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.851+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2810d3f0 con 0x7f6d28072500 2023-11-15T05:01:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.851+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2810d800 con 0x7f6d28072500 2023-11-15T05:01:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f6d1000f070 con 0x7f6d28072500 2023-11-15T05:01:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d28045ad0 con 0x7f6d28072500 2023-11-15T05:01:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d1f7fe700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 0x7f6d080321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6d10041370 con 0x7f6d28072500 2023-11-15T05:01:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d267fc700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 0x7f6d080321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.852+0000 7f6d267fc700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 0x7f6d080321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6d180096a0 tx=0x7f6d18006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:15.864 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:15.856+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d10019af0 con 0x7f6d28072500 2023-11-15T05:01:16.034 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.033+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 --> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d2810df30 con 0x7f6d0802fd10 2023-11-15T05:01:16.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.037+0000 7f6d1f7fe700 1 -- 172.21.15.86:0/2684725289 <== mgr.14142 v2:172.21.15.86:6800/1131374657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f6d2810df30 con 0x7f6d0802fd10 2023-11-15T05:01:16.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T05:01:16.039 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 msgr2=0x7f6d080321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:16.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 0x7f6d080321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6d180096a0 tx=0x7f6d18006b40).stop 2023-11-15T05:01:16.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 msgr2=0x7f6d2810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:16.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6d10011820 tx=0x7f6d10006ee0).stop 2023-11-15T05:01:16.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 shutdown_connections 2023-11-15T05:01:16.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f6d0802fd10 0x7f6d080321d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:16.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 --2- 172.21.15.86:0/2684725289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d28072500 0x7f6d2810cd00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:16.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 >> 172.21.15.86:0/2684725289 conn(0x7f6d280f74b0 msgr2=0x7f6d280f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:16.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 shutdown_connections 2023-11-15T05:01:16.041 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.039+0000 7f6d2da06700 1 -- 172.21.15.86:0/2684725289 wait complete. 2023-11-15T05:01:16.410 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:15.162409+0000 mgr.smithi086.hkfamo (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:16.411 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:15.162895+0000 mgr.smithi086.hkfamo (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T05:01:16.411 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:16.037409+0000 mon.smithi086 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:16.411 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:16.166499+0000 mon.smithi086 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi086.hkfamo(active, since 2s) 2023-11-15T05:01:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:Deploying alertmanager service with default placement... 2023-11-15T05:01:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T05:01:16.851 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.850+0000 7f6191002700 1 Processor -- start 2023-11-15T05:01:16.851 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.850+0000 7f6191002700 1 -- start start 2023-11-15T05:01:16.854 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.850+0000 7f6191002700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:16.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.850+0000 7f6191002700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c0fec90 con 0x7f618c0703f0 2023-11-15T05:01:16.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:16.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46458/0 (socket says 172.21.15.86:46458) 2023-11-15T05:01:16.855 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f618ad9d700 1 -- 172.21.15.86:0/4191414266 learned_addr learned my addr 172.21.15.86:0/4191414266 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:16.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f618ad9d700 1 -- 172.21.15.86:0/4191414266 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c0fedd0 con 0x7f618c0703f0 2023-11-15T05:01:16.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f618ad9d700 1 --2- 172.21.15.86:0/4191414266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f618000b5f0 tx=0x7f6180006ee0).ready entity=mon.0 client_cookie=8e3824f75727b378 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:16.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f6189d9b700 1 -- 172.21.15.86:0/4191414266 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f618000d600 con 0x7f618c0703f0 2023-11-15T05:01:16.856 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f6189d9b700 1 -- 172.21.15.86:0/4191414266 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f618000dbd0 con 0x7f618c0703f0 2023-11-15T05:01:16.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.851+0000 7f6189d9b700 1 -- 172.21.15.86:0/4191414266 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6180011c00 con 0x7f618c0703f0 2023-11-15T05:01:16.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 -- 172.21.15.86:0/4191414266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 msgr2=0x7f618c0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:16.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 --2- 172.21.15.86:0/4191414266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f618000b5f0 tx=0x7f6180006ee0).stop 2023-11-15T05:01:16.857 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 -- 172.21.15.86:0/4191414266 shutdown_connections 2023-11-15T05:01:16.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 --2- 172.21.15.86:0/4191414266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c0fe7a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:16.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 -- 172.21.15.86:0/4191414266 >> 172.21.15.86:0/4191414266 conn(0x7f618c0f6240 msgr2=0x7f618c0f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:16.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 -- 172.21.15.86:0/4191414266 shutdown_connections 2023-11-15T05:01:16.858 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 -- 172.21.15.86:0/4191414266 wait complete. 2023-11-15T05:01:16.859 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.852+0000 7f6191002700 1 Processor -- start 2023-11-15T05:01:16.859 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f6191002700 1 -- start start 2023-11-15T05:01:16.859 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f6191002700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:16.859 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f6191002700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c10ada0 con 0x7f618c0703f0 2023-11-15T05:01:16.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:16.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f618ad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46464/0 (socket says 172.21.15.86:46464) 2023-11-15T05:01:16.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f618ad9d700 1 -- 172.21.15.86:0/2047772007 learned_addr learned my addr 172.21.15.86:0/2047772007 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:16.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f618ad9d700 1 -- 172.21.15.86:0/2047772007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618000b200 con 0x7f618c0703f0 2023-11-15T05:01:16.860 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f618ad9d700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f618000e040 tx=0x7f6180006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:16.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.853+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f618000d9f0 con 0x7f618c0703f0 2023-11-15T05:01:16.861 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c10afa0 con 0x7f618c0703f0 2023-11-15T05:01:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f618001a9f0 con 0x7f618c0703f0 2023-11-15T05:01:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c10b3b0 con 0x7f618c0703f0 2023-11-15T05:01:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f618001b8c0 con 0x7f618c0703f0 2023-11-15T05:01:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f618000f070 con 0x7f618c0703f0 2023-11-15T05:01:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.854+0000 7f61737fe700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 0x7f6178032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.855+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f61800414c0 con 0x7f618c0703f0 2023-11-15T05:01:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.855+0000 7f618a59c700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 0x7f6178032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.855+0000 7f618a59c700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 0x7f6178032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.855+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c059100 con 0x7f618c0703f0 2023-11-15T05:01:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:16.859+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6180016070 con 0x7f618c0703f0 2023-11-15T05:01:17.037 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.036+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 --> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f618c104670 con 0x7f617802fe60 2023-11-15T05:01:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.039+0000 7f61737fe700 1 -- 172.21.15.86:0/2047772007 <== mgr.14142 v2:172.21.15.86:6800/1131374657 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f618c104670 con 0x7f617802fe60 2023-11-15T05:01:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T05:01:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 msgr2=0x7f6178032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 0x7f6178032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f617c0096a0 tx=0x7f617c006b40).stop 2023-11-15T05:01:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 msgr2=0x7f618c10a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f618000e040 tx=0x7f6180006ee0).stop 2023-11-15T05:01:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 shutdown_connections 2023-11-15T05:01:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f617802fe60 0x7f6178032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 --2- 172.21.15.86:0/2047772007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f618c0703f0 0x7f618c10a8b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 >> 172.21.15.86:0/2047772007 conn(0x7f618c0f6240 msgr2=0x7f618c0f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 shutdown_connections 2023-11-15T05:01:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.041+0000 7f6191002700 1 -- 172.21.15.86:0/2047772007 wait complete. 2023-11-15T05:01:17.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:16.034604+0000 mgr.smithi086.hkfamo (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:17.316 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:16.035274+0000 mgr.smithi086.hkfamo (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T05:01:17.316 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:17.039798+0000 mon.smithi086 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.86:0/2935143422' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:17.388 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T05:01:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.724+0000 7fb376bdf700 1 Processor -- start 2023-11-15T05:01:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.724+0000 7fb376bdf700 1 -- start start 2023-11-15T05:01:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.724+0000 7fb376bdf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.724+0000 7fb376bdf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700fedd0 con 0x7fb370070430 2023-11-15T05:01:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.724+0000 7fb37497b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:17.730 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb37497b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46468/0 (socket says 172.21.15.86:46468) 2023-11-15T05:01:17.730 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb37497b700 1 -- 172.21.15.86:0/796477044 learned_addr learned my addr 172.21.15.86:0/796477044 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:17.730 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb37497b700 1 -- 172.21.15.86:0/796477044 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3700fef10 con 0x7fb370070430 2023-11-15T05:01:17.730 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb37497b700 1 --2- 172.21.15.86:0/796477044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb36400b5f0 tx=0x7fb364006ee0).ready entity=mon.0 client_cookie=4dc40a8f4c13d259 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:17.731 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb36f7fe700 1 -- 172.21.15.86:0/796477044 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36400d600 con 0x7fb370070430 2023-11-15T05:01:17.731 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb36f7fe700 1 -- 172.21.15.86:0/796477044 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb36400dbd0 con 0x7fb370070430 2023-11-15T05:01:17.731 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.725+0000 7fb36f7fe700 1 -- 172.21.15.86:0/796477044 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb364011c00 con 0x7fb370070430 2023-11-15T05:01:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- 172.21.15.86:0/796477044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 msgr2=0x7fb3700fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 --2- 172.21.15.86:0/796477044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb36400b5f0 tx=0x7fb364006ee0).stop 2023-11-15T05:01:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- 172.21.15.86:0/796477044 shutdown_connections 2023-11-15T05:01:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 --2- 172.21.15.86:0/796477044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fe8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.732 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- 172.21.15.86:0/796477044 >> 172.21.15.86:0/796477044 conn(0x7fb3700f63c0 msgr2=0x7fb3700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:17.733 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- 172.21.15.86:0/796477044 shutdown_connections 2023-11-15T05:01:17.733 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- 172.21.15.86:0/796477044 wait complete. 2023-11-15T05:01:17.733 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 Processor -- start 2023-11-15T05:01:17.733 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.726+0000 7fb376bdf700 1 -- start start 2023-11-15T05:01:17.734 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb376bdf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:17.734 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb376bdf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700fa7e0 con 0x7fb370070430 2023-11-15T05:01:17.734 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb37497b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:17.735 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb37497b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46478/0 (socket says 172.21.15.86:46478) 2023-11-15T05:01:17.735 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb37497b700 1 -- 172.21.15.86:0/3644198868 learned_addr learned my addr 172.21.15.86:0/3644198868 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:17.735 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb37497b700 1 -- 172.21.15.86:0/3644198868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36400b200 con 0x7fb370070430 2023-11-15T05:01:17.736 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb37497b700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fb36400b190 tx=0x7fb364006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:17.736 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36401b410 con 0x7fb370070430 2023-11-15T05:01:17.736 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3700fa980 con 0x7fb370070430 2023-11-15T05:01:17.736 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb36401b9e0 con 0x7fb370070430 2023-11-15T05:01:17.737 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36401a880 con 0x7fb370070430 2023-11-15T05:01:17.737 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.727+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3700fad90 con 0x7fb370070430 2023-11-15T05:01:17.739 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.728+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb36400f070 con 0x7fb370070430 2023-11-15T05:01:17.739 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.728+0000 7fb36d7fa700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 0x7fb354032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:17.739 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.728+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb364041510 con 0x7fb370070430 2023-11-15T05:01:17.740 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.728+0000 7fb36ffff700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 0x7fb354032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:17.740 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.729+0000 7fb36ffff700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 0x7fb354032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:17.740 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.729+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb370059100 con 0x7fb370070430 2023-11-15T05:01:17.740 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.733+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb364010360 con 0x7fb370070430 2023-11-15T05:01:17.883 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.881+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb3701047f0 con 0x7fb370070430 2023-11-15T05:01:17.885 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.884+0000 7fb36d7fa700 1 -- 172.21.15.86:0/3644198868 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb364022b00 con 0x7fb370070430 2023-11-15T05:01:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 msgr2=0x7fb354032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 0x7fb354032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).stop 2023-11-15T05:01:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 msgr2=0x7fb3700fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fb36400b190 tx=0x7fb364006ee0).stop 2023-11-15T05:01:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 shutdown_connections 2023-11-15T05:01:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fb35402fe60 0x7fb354032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 --2- 172.21.15.86:0/3644198868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb370070430 0x7fb3700fc360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:17.889 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.886+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 >> 172.21.15.86:0/3644198868 conn(0x7fb3700f63c0 msgr2=0x7fb3700f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:17.890 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.887+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 shutdown_connections 2023-11-15T05:01:17.890 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:17.887+0000 7fb376bdf700 1 -- 172.21.15.86:0/3644198868 wait complete. 2023-11-15T05:01:18.230 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T05:01:18.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:17.037152+0000 mgr.smithi086.hkfamo (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:18.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:17.037653+0000 mgr.smithi086.hkfamo (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T05:01:18.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:17.886092+0000 mon.smithi086 (mon.0) 81 : audit [INF] from='client.? 172.21.15.86:0/3644198868' entity='client.admin' 2023-11-15T05:01:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91c903700 1 Processor -- start 2023-11-15T05:01:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91c903700 1 -- start start 2023-11-15T05:01:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91c903700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91c903700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180fee10 con 0x7fd9180703f0 2023-11-15T05:01:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39998/0 (socket says 172.21.15.86:39998) 2023-11-15T05:01:18.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91659c700 1 -- 172.21.15.86:0/3030067564 learned_addr learned my addr 172.21.15.86:0/3030067564 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:18.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91659c700 1 -- 172.21.15.86:0/3030067564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180fef50 con 0x7fd9180703f0 2023-11-15T05:01:18.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.597+0000 7fd91659c700 1 --2- 172.21.15.86:0/3030067564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd90c00b5f0 tx=0x7fd90c006ee0).ready entity=mon.0 client_cookie=3349ff8831272d38 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:18.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91559a700 1 -- 172.21.15.86:0/3030067564 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90c00d600 con 0x7fd9180703f0 2023-11-15T05:01:18.602 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91559a700 1 -- 172.21.15.86:0/3030067564 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd90c00dbd0 con 0x7fd9180703f0 2023-11-15T05:01:18.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91559a700 1 -- 172.21.15.86:0/3030067564 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90c011c00 con 0x7fd9180703f0 2023-11-15T05:01:18.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 -- 172.21.15.86:0/3030067564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 msgr2=0x7fd9180fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:18.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 --2- 172.21.15.86:0/3030067564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd90c00b5f0 tx=0x7fd90c006ee0).stop 2023-11-15T05:01:18.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 -- 172.21.15.86:0/3030067564 shutdown_connections 2023-11-15T05:01:18.603 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 --2- 172.21.15.86:0/3030067564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd9180fe920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:18.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 -- 172.21.15.86:0/3030067564 >> 172.21.15.86:0/3030067564 conn(0x7fd9180f63c0 msgr2=0x7fd9180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:18.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 -- 172.21.15.86:0/3030067564 shutdown_connections 2023-11-15T05:01:18.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.598+0000 7fd91c903700 1 -- 172.21.15.86:0/3030067564 wait complete. 2023-11-15T05:01:18.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91c903700 1 Processor -- start 2023-11-15T05:01:18.604 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91c903700 1 -- start start 2023-11-15T05:01:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91c903700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91c903700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91810af10 con 0x7fd9180703f0 2023-11-15T05:01:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40010/0 (socket says 172.21.15.86:40010) 2023-11-15T05:01:18.605 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91659c700 1 -- 172.21.15.86:0/1984482966 learned_addr learned my addr 172.21.15.86:0/1984482966 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:18.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91659c700 1 -- 172.21.15.86:0/1984482966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c00b200 con 0x7fd9180703f0 2023-11-15T05:01:18.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.599+0000 7fd91659c700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd90c00e040 tx=0x7fd90c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:18.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90c00d9f0 con 0x7fd9180703f0 2023-11-15T05:01:18.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd90c01a9f0 con 0x7fd9180703f0 2023-11-15T05:01:18.607 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91810b110 con 0x7fd9180703f0 2023-11-15T05:01:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd90c01b8c0 con 0x7fd9180703f0 2023-11-15T05:01:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91810b520 con 0x7fd9180703f0 2023-11-15T05:01:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.600+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd90c00f070 con 0x7fd9180703f0 2023-11-15T05:01:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.601+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd918045ad0 con 0x7fd9180703f0 2023-11-15T05:01:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.601+0000 7fd906ffd700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 0x7fd8fc032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.601+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd90c041570 con 0x7fd9180703f0 2023-11-15T05:01:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.601+0000 7fd915d9b700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 0x7fd8fc032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.601+0000 7fd915d9b700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 0x7fd8fc032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd9080096a0 tx=0x7fd908006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.605+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd90c0224f0 con 0x7fd9180703f0 2023-11-15T05:01:18.754 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.753+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd9180ffbd0 con 0x7fd9180703f0 2023-11-15T05:01:18.761 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.760+0000 7fd906ffd700 1 -- 172.21.15.86:0/1984482966 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd90c006a70 con 0x7fd9180703f0 2023-11-15T05:01:18.763 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 msgr2=0x7fd8fc032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 0x7fd8fc032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd9080096a0 tx=0x7fd908006b40).stop 2023-11-15T05:01:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 msgr2=0x7fd91810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd90c00e040 tx=0x7fd90c006ee0).stop 2023-11-15T05:01:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 shutdown_connections 2023-11-15T05:01:18.764 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fd8fc02fe60 0x7fd8fc032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:18.765 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 --2- 172.21.15.86:0/1984482966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9180703f0 0x7fd91810aa20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:18.765 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 >> 172.21.15.86:0/1984482966 conn(0x7fd9180f63c0 msgr2=0x7fd9180f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:18.766 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 shutdown_connections 2023-11-15T05:01:18.766 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:18.762+0000 7fd91c903700 1 -- 172.21.15.86:0/1984482966 wait complete. 2023-11-15T05:01:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:Enabling the dashboard module... 2023-11-15T05:01:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T05:01:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.434+0000 7f8070ee6700 1 Processor -- start 2023-11-15T05:01:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.434+0000 7f8070ee6700 1 -- start start 2023-11-15T05:01:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.434+0000 7f8070ee6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.434+0000 7f8070ee6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0fede0 con 0x7f806c0704a0 2023-11-15T05:01:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40022/0 (socket says 172.21.15.86:40022) 2023-11-15T05:01:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806a59c700 1 -- 172.21.15.86:0/4287649287 learned_addr learned my addr 172.21.15.86:0/4287649287 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806a59c700 1 -- 172.21.15.86:0/4287649287 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c0fef20 con 0x7f806c0704a0 2023-11-15T05:01:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806a59c700 1 --2- 172.21.15.86:0/4287649287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f806000b5f0 tx=0x7f8060006ee0).ready entity=mon.0 client_cookie=c87b90a62ec0201e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806959a700 1 -- 172.21.15.86:0/4287649287 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806000d600 con 0x7f806c0704a0 2023-11-15T05:01:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806959a700 1 -- 172.21.15.86:0/4287649287 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f806000dbd0 con 0x7f806c0704a0 2023-11-15T05:01:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.435+0000 7f806959a700 1 -- 172.21.15.86:0/4287649287 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8060011c00 con 0x7f806c0704a0 2023-11-15T05:01:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 -- 172.21.15.86:0/4287649287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 msgr2=0x7f806c0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 --2- 172.21.15.86:0/4287649287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f806000b5f0 tx=0x7f8060006ee0).stop 2023-11-15T05:01:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 -- 172.21.15.86:0/4287649287 shutdown_connections 2023-11-15T05:01:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 --2- 172.21.15.86:0/4287649287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c0fe8f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 -- 172.21.15.86:0/4287649287 >> 172.21.15.86:0/4287649287 conn(0x7f806c0f63b0 msgr2=0x7f806c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 -- 172.21.15.86:0/4287649287 shutdown_connections 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.436+0000 7f8070ee6700 1 -- 172.21.15.86:0/4287649287 wait complete. 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f8070ee6700 1 Processor -- start 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f8070ee6700 1 -- start start 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f8070ee6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f8070ee6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c10aee0 con 0x7f806c0704a0 2023-11-15T05:01:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f806a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:19.446 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f806a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40032/0 (socket says 172.21.15.86:40032) 2023-11-15T05:01:19.446 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f806a59c700 1 -- 172.21.15.86:0/1398615186 learned_addr learned my addr 172.21.15.86:0/1398615186 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:19.446 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f806a59c700 1 -- 172.21.15.86:0/1398615186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806000b200 con 0x7f806c0704a0 2023-11-15T05:01:19.446 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f806a59c700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f806000b190 tx=0x7f8060006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:19.446 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806001b410 con 0x7f806c0704a0 2023-11-15T05:01:19.447 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f806001b9e0 con 0x7f806c0704a0 2023-11-15T05:01:19.447 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c10b0e0 con 0x7f806c0704a0 2023-11-15T05:01:19.447 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.437+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806001a880 con 0x7f806c0704a0 2023-11-15T05:01:19.447 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.438+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c10b4f0 con 0x7f806c0704a0 2023-11-15T05:01:19.447 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.438+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f806000f070 con 0x7f806c0704a0 2023-11-15T05:01:19.448 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.438+0000 7f805affd700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 0x7f8050032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:19.448 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.438+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c045ad0 con 0x7f806c0704a0 2023-11-15T05:01:19.448 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.439+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8060041330 con 0x7f806c0704a0 2023-11-15T05:01:19.448 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.439+0000 7f8069d9b700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 0x7f8050032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:19.448 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.439+0000 7f8069d9b700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 0x7f8050032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f805c0096a0 tx=0x7f805c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:19.449 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.443+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8060019950 con 0x7f806c0704a0 2023-11-15T05:01:19.631 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.630+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f806c0ffba0 con 0x7f806c0704a0 2023-11-15T05:01:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.768+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f806001bb50 con 0x7f806c0704a0 2023-11-15T05:01:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.769+0000 7f805affd700 1 -- 172.21.15.86:0/1398615186 <== mon.0 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8060008900 con 0x7f806c0704a0 2023-11-15T05:01:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.770+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 msgr2=0x7f8050032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.770+0000 7f8070ee6700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 0x7f8050032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f805c0096a0 tx=0x7f805c006b40).stop 2023-11-15T05:01:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.770+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 msgr2=0x7f806c10a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.770+0000 7f8070ee6700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f806000b190 tx=0x7f8060006ee0).stop 2023-11-15T05:01:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 shutdown_connections 2023-11-15T05:01:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f805002feb0 0x7f8050032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:19.779 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 --2- 172.21.15.86:0/1398615186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f806c0704a0 0x7f806c10a9f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:19.779 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 >> 172.21.15.86:0/1398615186 conn(0x7f806c0f63b0 msgr2=0x7f806c0f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:19.779 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 shutdown_connections 2023-11-15T05:01:19.780 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:19.771+0000 7f8070ee6700 1 -- 172.21.15.86:0/1398615186 wait complete. 2023-11-15T05:01:20.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:18.761482+0000 mon.smithi086 (mon.0) 82 : audit [INF] from='client.? 172.21.15.86:0/1984482966' entity='client.admin' 2023-11-15T05:01:20.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:19.631833+0000 mon.smithi086 (mon.0) 83 : audit [INF] from='client.? 172.21.15.86:0/1398615186' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T05:01:20.113 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T05:01:20.483 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7343587700 1 Processor -- start 2023-11-15T05:01:20.484 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7343587700 1 -- start start 2023-11-15T05:01:20.484 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7343587700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:20.484 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7343587700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0fee10 con 0x7f733c0703f0 2023-11-15T05:01:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7341323700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7341323700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40056/0 (socket says 172.21.15.86:40056) 2023-11-15T05:01:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.480+0000 7f7341323700 1 -- 172.21.15.86:0/945282159 learned_addr learned my addr 172.21.15.86:0/945282159 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f7341323700 1 -- 172.21.15.86:0/945282159 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0fef50 con 0x7f733c0703f0 2023-11-15T05:01:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f7341323700 1 --2- 172.21.15.86:0/945282159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f733800b5f0 tx=0x7f7338006ee0).ready entity=mon.0 client_cookie=22537b52a43fd2c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:20.486 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f732bfff700 1 -- 172.21.15.86:0/945282159 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f733800d600 con 0x7f733c0703f0 2023-11-15T05:01:20.486 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f732bfff700 1 -- 172.21.15.86:0/945282159 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f733800dbd0 con 0x7f733c0703f0 2023-11-15T05:01:20.486 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f732bfff700 1 -- 172.21.15.86:0/945282159 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7338011c00 con 0x7f733c0703f0 2023-11-15T05:01:20.486 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f7343587700 1 -- 172.21.15.86:0/945282159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 msgr2=0x7f733c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:20.487 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.481+0000 7f7343587700 1 --2- 172.21.15.86:0/945282159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f733800b5f0 tx=0x7f7338006ee0).stop 2023-11-15T05:01:20.487 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 -- 172.21.15.86:0/945282159 shutdown_connections 2023-11-15T05:01:20.487 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 --2- 172.21.15.86:0/945282159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c0fe920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:20.487 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 -- 172.21.15.86:0/945282159 >> 172.21.15.86:0/945282159 conn(0x7f733c0f63c0 msgr2=0x7f733c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:20.487 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 -- 172.21.15.86:0/945282159 shutdown_connections 2023-11-15T05:01:20.488 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 -- 172.21.15.86:0/945282159 wait complete. 2023-11-15T05:01:20.488 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 Processor -- start 2023-11-15T05:01:20.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.482+0000 7f7343587700 1 -- start start 2023-11-15T05:01:20.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7343587700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:20.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7343587700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c10af40 con 0x7f733c0703f0 2023-11-15T05:01:20.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7341323700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:20.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7341323700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40066/0 (socket says 172.21.15.86:40066) 2023-11-15T05:01:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7341323700 1 -- 172.21.15.86:0/1632857249 learned_addr learned my addr 172.21.15.86:0/1632857249 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7341323700 1 -- 172.21.15.86:0/1632857249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733800b200 con 0x7f733c0703f0 2023-11-15T05:01:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7341323700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f733800e040 tx=0x7f7338006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f733800d9f0 con 0x7f733c0703f0 2023-11-15T05:01:20.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f733801a9f0 con 0x7f733c0703f0 2023-11-15T05:01:20.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f733801b8c0 con 0x7f733c0703f0 2023-11-15T05:01:20.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.483+0000 7f7343587700 1 -- 172.21.15.86:0/1632857249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c10b140 con 0x7f733c0703f0 2023-11-15T05:01:20.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.484+0000 7f7343587700 1 -- 172.21.15.86:0/1632857249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c10b550 con 0x7f733c0703f0 2023-11-15T05:01:20.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.484+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f733800f070 con 0x7f733c0703f0 2023-11-15T05:01:20.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.485+0000 7f7329ffb700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 0x7f73240323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:20.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.485+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7318004fa0 con 0x7f733c0703f0 2023-11-15T05:01:20.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.485+0000 7f7340b22700 1 -- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 msgr2=0x7f73240323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:20.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.485+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7338041640 con 0x7f733c0703f0 2023-11-15T05:01:20.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.485+0000 7f7340b22700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 0x7f73240323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:01:20.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.489+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7338022900 con 0x7f733c0703f0 2023-11-15T05:01:20.687 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.686+0000 7f7340b22700 1 -- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 msgr2=0x7f73240323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:20.687 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.686+0000 7f7340b22700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 0x7f73240323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:01:20.694 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.692+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7318005e80 con 0x7f733c0703f0 2023-11-15T05:01:20.695 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.692+0000 7f7329ffb700 1 -- 172.21.15.86:0/1632857249 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f7338016070 con 0x7f733c0703f0 2023-11-15T05:01:20.695 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:01:20.695 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "epoch": 12, 2023-11-15T05:01:20.695 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:01:20.696 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "active_name": "smithi086.hkfamo", 2023-11-15T05:01:20.696 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T05:01:20.696 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:01:20.697 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 msgr2=0x7f73240323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:01:20.698 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 0x7f73240323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:20.698 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 msgr2=0x7f733c10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:20.698 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f733800e040 tx=0x7f7338006ee0).stop 2023-11-15T05:01:20.698 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 shutdown_connections 2023-11-15T05:01:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7f732402ff00 0x7f73240323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 --2- 172.21.15.86:0/1632857249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f733c0703f0 0x7f733c10aa50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 >> 172.21.15.86:0/1632857249 conn(0x7f733c0f63c0 msgr2=0x7f733c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 shutdown_connections 2023-11-15T05:01:20.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:20.694+0000 7f73237fe700 1 -- 172.21.15.86:0/1632857249 wait complete. 2023-11-15T05:01:20.982 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:19.770447+0000 mon.smithi086 (mon.0) 84 : audit [INF] from='client.? 172.21.15.86:0/1398615186' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T05:01:20.982 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:19.770504+0000 mon.smithi086 (mon.0) 85 : cluster [DBG] mgrmap e12: smithi086.hkfamo(active, since 6s) 2023-11-15T05:01:20.982 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:20.693925+0000 mon.smithi086 (mon.0) 86 : audit [DBG] from='client.? 172.21.15.86:0/1632857249' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T05:01:21.081 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for the mgr to restart... 2023-11-15T05:01:21.081 INFO:teuthology.orchestra.run.smithi086.stderr:Waiting for mgr epoch 12... 2023-11-15T05:01:21.082 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T05:01:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fe3a2700 1 Processor -- start 2023-11-15T05:01:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fe3a2700 1 -- start start 2023-11-15T05:01:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fe3a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fe3a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80faa70 con 0x7fe9f80fa160 2023-11-15T05:01:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fd3a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fd3a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40074/0 (socket says 172.21.15.86:40074) 2023-11-15T05:01:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.696+0000 7fe9fd3a0700 1 -- 172.21.15.86:0/3673769778 learned_addr learned my addr 172.21.15.86:0/3673769778 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.697+0000 7fe9fd3a0700 1 -- 172.21.15.86:0/3673769778 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f80fabb0 con 0x7fe9f80fa160 2023-11-15T05:01:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.697+0000 7fe9fd3a0700 1 --2- 172.21.15.86:0/3673769778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe9f400b5f0 tx=0x7fe9f4006ee0).ready entity=mon.0 client_cookie=17d7c5a84e84b651 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.697+0000 7fe9e7fff700 1 -- 172.21.15.86:0/3673769778 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f400d600 con 0x7fe9f80fa160 2023-11-15T05:01:21.701 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.697+0000 7fe9e7fff700 1 -- 172.21.15.86:0/3673769778 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe9f400dbd0 con 0x7fe9f80fa160 2023-11-15T05:01:21.701 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.697+0000 7fe9e7fff700 1 -- 172.21.15.86:0/3673769778 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f4011c00 con 0x7fe9f80fa160 2023-11-15T05:01:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.698+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/3673769778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 msgr2=0x7fe9f80fa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.698+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/3673769778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe9f400b5f0 tx=0x7fe9f4006ee0).stop 2023-11-15T05:01:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.698+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/3673769778 shutdown_connections 2023-11-15T05:01:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.698+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/3673769778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f80fa580 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.698+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/3673769778 >> 172.21.15.86:0/3673769778 conn(0x7fe9f80f59a0 msgr2=0x7fe9f80f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/3673769778 shutdown_connections 2023-11-15T05:01:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/3673769778 wait complete. 2023-11-15T05:01:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 Processor -- start 2023-11-15T05:01:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 -- start start 2023-11-15T05:01:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.699+0000 7fe9fe3a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f80f7800 con 0x7fe9f80fa160 2023-11-15T05:01:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fd3a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fd3a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40080/0 (socket says 172.21.15.86:40080) 2023-11-15T05:01:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fd3a0700 1 -- 172.21.15.86:0/2668106194 learned_addr learned my addr 172.21.15.86:0/2668106194 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fd3a0700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f400b200 con 0x7fe9f80fa160 2023-11-15T05:01:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fd3a0700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe9f400e040 tx=0x7fe9f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f401b410 con 0x7fe9f80fa160 2023-11-15T05:01:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe9f401b9e0 con 0x7fe9f80fa160 2023-11-15T05:01:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f401a770 con 0x7fe9f80fa160 2023-11-15T05:01:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f80f7a00 con 0x7fe9f80fa160 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.700+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f810dc70 con 0x7fe9f80fa160 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fe9f400f070 con 0x7fe9f80fa160 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9e5ffb700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9f8059100 con 0x7fe9e002ff00 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe9f4041100 con 0x7fe9f80fa160 2023-11-15T05:01:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9fcb9f700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.701+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:01:21.903 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.902+0000 7fe9fcb9f700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:21.904 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:21.902+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:01:22.304 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:22.303+0000 7fe9fcb9f700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:22.304 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:22.303+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:01:23.105 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:23.104+0000 7fe9fcb9f700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:23.105 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:23.104+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:01:24.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:24.705+0000 7fe9fcb9f700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:6800/1131374657 2023-11-15T05:01:24.707 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:24.705+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T05:01:24.792 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:24.790+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fe9f401bb50 con 0x7fe9f80fa160 2023-11-15T05:01:24.792 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:24.790+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 msgr2=0x7fe9e00323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:01:24.792 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:24.790+0000 7fe9e5ffb700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:24.740227+0000 mon.smithi086 (mon.0) 87 : cluster [INF] Active manager daemon smithi086.hkfamo restarted 2023-11-15T05:01:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:24.740616+0000 mon.smithi086 (mon.0) 88 : cluster [INF] Activating manager daemon smithi086.hkfamo 2023-11-15T05:01:25.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:24.742264+0000 mon.smithi086 (mon.0) 89 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T05:01:25.807 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.804+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fe9f4041e80 con 0x7fe9f80fa160 2023-11-15T05:01:25.807 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.804+0000 7fe9e5ffb700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 0x7fe9e0035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:25.807 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.804+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9f8059100 con 0x7fe9e00339e0 2023-11-15T05:01:25.807 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.804+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 0x7fe9e0035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:25.808 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.805+0000 7fe9fcb9f700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 0x7fe9e0035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe9e8006fd0 tx=0x7fe9e8006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:25.808 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.805+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fe9f8059100 con 0x7fe9e00339e0 2023-11-15T05:01:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.811+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe9f8045ad0 con 0x7fe9e00339e0 2023-11-15T05:01:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.811+0000 7fe9e5ffb700 1 -- 172.21.15.86:0/2668106194 <== mgr.14162 v2:172.21.15.86:6800/432331256 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe9f8045ad0 con 0x7fe9e00339e0 2023-11-15T05:01:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: { 2023-11-15T05:01:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-11-15T05:01:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T05:01:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: } 2023-11-15T05:01:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 msgr2=0x7fe9e0035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 0x7fe9e0035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe9e8006fd0 tx=0x7fe9e8006d70).stop 2023-11-15T05:01:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 msgr2=0x7fe9f810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:25.817 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe9f400e040 tx=0x7fe9f4006ee0).stop 2023-11-15T05:01:25.817 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 shutdown_connections 2023-11-15T05:01:25.817 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe9e00339e0 0x7fe9e0035dd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:25.817 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f80fa160 0x7fe9f810d5c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:25.817 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 --2- 172.21.15.86:0/2668106194 >> [v2:172.21.15.86:6800/1131374657,v1:172.21.15.86:6801/1131374657] conn(0x7fe9e002ff00 0x7fe9e00323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:01:25.818 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 >> 172.21.15.86:0/2668106194 conn(0x7fe9f80f59a0 msgr2=0x7fe9f8100760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:25.818 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 shutdown_connections 2023-11-15T05:01:25.818 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:25.812+0000 7fe9fe3a2700 1 -- 172.21.15.86:0/2668106194 wait complete. 2023-11-15T05:01:26.098 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:24.792946+0000 mon.smithi086 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi086.hkfamo(active, starting, since 0.0524142s) 2023-11-15T05:01:26.098 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.794172+0000 mon.smithi086 (mon.0) 91 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2023-11-15T05:01:26.098 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.794850+0000 mon.smithi086 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mgr metadata", "who": "smithi086.hkfamo", "id": "smithi086.hkfamo"}]: dispatch 2023-11-15T05:01:26.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.795585+0000 mon.smithi086 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:01:26.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.795755+0000 mon.smithi086 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:01:26.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.795913+0000 mon.smithi086 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:01:26.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:24.809746+0000 mon.smithi086 (mon.0) 96 : cluster [INF] Manager daemon smithi086.hkfamo is now available 2023-11-15T05:01:26.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.905416+0000 mon.smithi086 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:01:26.100 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.908020+0000 mon.smithi086 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:01:26.100 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.922340+0000 mon.smithi086 (mon.0) 99 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:01:26.100 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:24.931188+0000 mon.smithi086 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:01:26.129 INFO:teuthology.orchestra.run.smithi086.stderr:mgr epoch 12 is available 2023-11-15T05:01:26.130 INFO:teuthology.orchestra.run.smithi086.stderr:Generating a dashboard self-signed certificate... 2023-11-15T05:01:26.130 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T05:01:26.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f54214cd700 1 Processor -- start 2023-11-15T05:01:26.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f54214cd700 1 -- start start 2023-11-15T05:01:26.459 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f54214cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:26.460 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f54214cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0fb4a0 con 0x7f541c0fab90 2023-11-15T05:01:26.460 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f541affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:26.460 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f541affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40142/0 (socket says 172.21.15.86:40142) 2023-11-15T05:01:26.463 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f541affd700 1 -- 172.21.15.86:0/1387600861 learned_addr learned my addr 172.21.15.86:0/1387600861 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:26.464 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f541affd700 1 -- 172.21.15.86:0/1387600861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c0fb5e0 con 0x7f541c0fab90 2023-11-15T05:01:26.464 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.457+0000 7f541affd700 1 --2- 172.21.15.86:0/1387600861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f540c00b870 tx=0x7f540c006f70).ready entity=mon.0 client_cookie=bf53151c51d4812e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:26.464 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f5419ffb700 1 -- 172.21.15.86:0/1387600861 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c00da20 con 0x7f541c0fab90 2023-11-15T05:01:26.464 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f5419ffb700 1 -- 172.21.15.86:0/1387600861 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f540c010470 con 0x7f541c0fab90 2023-11-15T05:01:26.464 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f5419ffb700 1 -- 172.21.15.86:0/1387600861 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c00f5b0 con 0x7f541c0fab90 2023-11-15T05:01:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 -- 172.21.15.86:0/1387600861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 msgr2=0x7f541c0fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 --2- 172.21.15.86:0/1387600861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f540c00b870 tx=0x7f540c006f70).stop 2023-11-15T05:01:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 -- 172.21.15.86:0/1387600861 shutdown_connections 2023-11-15T05:01:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 --2- 172.21.15.86:0/1387600861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c0fafb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 -- 172.21.15.86:0/1387600861 >> 172.21.15.86:0/1387600861 conn(0x7f541c0f63b0 msgr2=0x7f541c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:26.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 -- 172.21.15.86:0/1387600861 shutdown_connections 2023-11-15T05:01:26.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.458+0000 7f54214cd700 1 -- 172.21.15.86:0/1387600861 wait complete. 2023-11-15T05:01:26.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f54214cd700 1 Processor -- start 2023-11-15T05:01:26.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f54214cd700 1 -- start start 2023-11-15T05:01:26.466 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f54214cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f54214cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c10d030 con 0x7f541c0fab90 2023-11-15T05:01:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f541affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f541affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40146/0 (socket says 172.21.15.86:40146) 2023-11-15T05:01:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f541affd700 1 -- 172.21.15.86:0/2454144050 learned_addr learned my addr 172.21.15.86:0/2454144050 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f541affd700 1 -- 172.21.15.86:0/2454144050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c00b480 con 0x7f541c0fab90 2023-11-15T05:01:26.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.459+0000 7f541affd700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f540c006a40 tx=0x7f540c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:26.468 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c00de00 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c10d230 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f540c00fef0 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c019e90 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c10d640 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.460+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f540c016030 con 0x7f541c0fab90 2023-11-15T05:01:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.461+0000 7f54037fe700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 0x7f5408032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:26.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.461+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f540c0466e0 con 0x7f541c0fab90 2023-11-15T05:01:26.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.461+0000 7f541a7fc700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 0x7f5408032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:26.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.461+0000 7f541a7fc700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 0x7f5408032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54100096a0 tx=0x7f5410006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:26.470 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.461+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c045ad0 con 0x7f541c0fab90 2023-11-15T05:01:26.471 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.465+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f540c014070 con 0x7f541c0fab90 2023-11-15T05:01:26.620 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.618+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f541c1026f0 con 0x7f540802fd50 2023-11-15T05:01:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.699+0000 7f54037fe700 1 -- 172.21.15.86:0/2454144050 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f541c1026f0 con 0x7f540802fd50 2023-11-15T05:01:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T05:01:26.703 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 msgr2=0x7f5408032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:26.703 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 0x7f5408032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54100096a0 tx=0x7f5410006b40).stop 2023-11-15T05:01:26.703 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 msgr2=0x7f541c10cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:26.704 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f540c006a40 tx=0x7f540c006f70).stop 2023-11-15T05:01:26.704 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 shutdown_connections 2023-11-15T05:01:26.704 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540802fd50 0x7f5408032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:26.704 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 --2- 172.21.15.86:0/2454144050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f541c0fab90 0x7f541c10cb40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:26.704 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 >> 172.21.15.86:0/2454144050 conn(0x7f541c0f63b0 msgr2=0x7f541c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:26.705 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 shutdown_connections 2023-11-15T05:01:26.705 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:26.701+0000 7f54214cd700 1 -- 172.21.15.86:0/2454144050 wait complete. 2023-11-15T05:01:26.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:26 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:25.806917+0000 mon.smithi086 (mon.0) 101 : cluster [DBG] mgrmap e14: smithi086.hkfamo(active, since 1.06639s) 2023-11-15T05:01:26.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:26 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:26.693666+0000 mon.smithi086 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:26.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:26 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:26.701068+0000 mon.smithi086 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:27.045 INFO:teuthology.orchestra.run.smithi086.stderr:Creating initial admin user... 2023-11-15T05:01:27.046 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3yt89fdm:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T05:01:27.411 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f106649a700 1 Processor -- start 2023-11-15T05:01:27.411 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f106649a700 1 -- start start 2023-11-15T05:01:27.415 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f106649a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:27.415 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f106649a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600fb490 con 0x7f10600fab80 2023-11-15T05:01:27.416 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f105ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:27.416 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f105ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40162/0 (socket says 172.21.15.86:40162) 2023-11-15T05:01:27.416 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f105ffff700 1 -- 172.21.15.86:0/1299162409 learned_addr learned my addr 172.21.15.86:0/1299162409 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:27.416 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f105ffff700 1 -- 172.21.15.86:0/1299162409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600fb5d0 con 0x7f10600fab80 2023-11-15T05:01:27.416 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.410+0000 7f105ffff700 1 --2- 172.21.15.86:0/1299162409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f105000b5f0 tx=0x7f1050006ee0).ready entity=mon.0 client_cookie=2eb66114546640a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:27.417 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f105effd700 1 -- 172.21.15.86:0/1299162409 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105000d600 con 0x7f10600fab80 2023-11-15T05:01:27.417 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f105effd700 1 -- 172.21.15.86:0/1299162409 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f105000dbd0 con 0x7f10600fab80 2023-11-15T05:01:27.417 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f105effd700 1 -- 172.21.15.86:0/1299162409 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1050011c00 con 0x7f10600fab80 2023-11-15T05:01:27.417 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 -- 172.21.15.86:0/1299162409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 msgr2=0x7f10600fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:27.417 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 --2- 172.21.15.86:0/1299162409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f105000b5f0 tx=0x7f1050006ee0).stop 2023-11-15T05:01:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 -- 172.21.15.86:0/1299162409 shutdown_connections 2023-11-15T05:01:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 --2- 172.21.15.86:0/1299162409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f10600fafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 -- 172.21.15.86:0/1299162409 >> 172.21.15.86:0/1299162409 conn(0x7f10600f63c0 msgr2=0x7f10600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 -- 172.21.15.86:0/1299162409 shutdown_connections 2023-11-15T05:01:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.411+0000 7f106649a700 1 -- 172.21.15.86:0/1299162409 wait complete. 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f106649a700 1 Processor -- start 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f106649a700 1 -- start start 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f106649a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f106649a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600704b0 con 0x7f10600fab80 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f105ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:27.419 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f105ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40170/0 (socket says 172.21.15.86:40170) 2023-11-15T05:01:27.420 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f105ffff700 1 -- 172.21.15.86:0/171943260 learned_addr learned my addr 172.21.15.86:0/171943260 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:27.420 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.412+0000 7f105ffff700 1 -- 172.21.15.86:0/171943260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105000b200 con 0x7f10600fab80 2023-11-15T05:01:27.420 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f105ffff700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f105000b1d0 tx=0x7f1050006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:27.420 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105001b410 con 0x7f10600fab80 2023-11-15T05:01:27.420 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f105001b9e0 con 0x7f10600fab80 2023-11-15T05:01:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10600706b0 con 0x7f10600fab80 2023-11-15T05:01:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105001a840 con 0x7f10600fab80 2023-11-15T05:01:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.413+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106010dd70 con 0x7f10600fab80 2023-11-15T05:01:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f105000f070 con 0x7f10600fab80 2023-11-15T05:01:27.421 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f105cff9700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 0x7f104c032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:27.422 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1050041300 con 0x7f10600fab80 2023-11-15T05:01:27.422 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f105f7fe700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 0x7f104c032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:27.422 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f105f7fe700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 0x7f104c032260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:27.422 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.414+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1060045ad0 con 0x7f10600fab80 2023-11-15T05:01:27.423 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.418+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1050016070 con 0x7f10600fab80 2023-11-15T05:01:27.584 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.583+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1060058f20 con 0x7f104c02fda0 2023-11-15T05:01:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.700+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f10500257e0 con 0x7f10600fab80 2023-11-15T05:01:27.800 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.798+0000 7f105cff9700 1 -- 172.21.15.86:0/171943260 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1060058f20 con 0x7f104c02fda0 2023-11-15T05:01:27.800 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$zhiTixdx/MXZLoH3Rvzgo.YY/L01AwtdmeW7B4/Pk6DJOAfy2XBYe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700024487, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T05:01:27.802 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 msgr2=0x7f104c032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:27.802 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 0x7f104c032260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).stop 2023-11-15T05:01:27.802 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 msgr2=0x7f106010d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f105000b1d0 tx=0x7f1050006ee0).stop 2023-11-15T05:01:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 shutdown_connections 2023-11-15T05:01:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f104c02fda0 0x7f104c032260 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 --2- 172.21.15.86:0/171943260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10600fab80 0x7f106010d6c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 >> 172.21.15.86:0/171943260 conn(0x7f10600f63c0 msgr2=0x7f10600f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:27.804 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 shutdown_connections 2023-11-15T05:01:27.804 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:27.800+0000 7f106649a700 1 -- 172.21.15.86:0/171943260 wait complete. 2023-11-15T05:01:28.064 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:25.806809+0000 mgr.smithi086.hkfamo (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T05:01:28.064 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:25.813213+0000 mgr.smithi086.hkfamo (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T05:01:28.064 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:26.620643+0000 mgr.smithi086.hkfamo (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:28.064 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:27.703036+0000 mon.smithi086 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi086.hkfamo(active, since 2s) 2023-11-15T05:01:28.065 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:27.799909+0000 mon.smithi086 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:Fetching dashboard port number... 2023-11-15T05:01:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpbsrp86jc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpaijg76ia:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T05:01:28.488 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.483+0000 7f24661fe700 1 Processor -- start 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.483+0000 7f24661fe700 1 -- start start 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f24661fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:28.489 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f24661fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600fedd0 con 0x7f2460070430 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47388/0 (socket says 172.21.15.86:47388) 2023-11-15T05:01:28.490 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245f7fe700 1 -- 172.21.15.86:0/2377430698 learned_addr learned my addr 172.21.15.86:0/2377430698 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:28.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245f7fe700 1 -- 172.21.15.86:0/2377430698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600fef10 con 0x7f2460070430 2023-11-15T05:01:28.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245f7fe700 1 --2- 172.21.15.86:0/2377430698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f245000b5f0 tx=0x7f2450006ee0).ready entity=mon.0 client_cookie=ae0656ef8c546f6d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:28.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245e7fc700 1 -- 172.21.15.86:0/2377430698 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f245000d600 con 0x7f2460070430 2023-11-15T05:01:28.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245e7fc700 1 -- 172.21.15.86:0/2377430698 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f245000dbd0 con 0x7f2460070430 2023-11-15T05:01:28.491 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.484+0000 7f245e7fc700 1 -- 172.21.15.86:0/2377430698 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2450011c00 con 0x7f2460070430 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 -- 172.21.15.86:0/2377430698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 msgr2=0x7f24600fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 --2- 172.21.15.86:0/2377430698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f245000b5f0 tx=0x7f2450006ee0).stop 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 -- 172.21.15.86:0/2377430698 shutdown_connections 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 --2- 172.21.15.86:0/2377430698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f24600fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 -- 172.21.15.86:0/2377430698 >> 172.21.15.86:0/2377430698 conn(0x7f24600f63c0 msgr2=0x7f24600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 -- 172.21.15.86:0/2377430698 shutdown_connections 2023-11-15T05:01:28.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 -- 172.21.15.86:0/2377430698 wait complete. 2023-11-15T05:01:28.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.485+0000 7f24661fe700 1 Processor -- start 2023-11-15T05:01:28.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f24661fe700 1 -- start start 2023-11-15T05:01:28.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f24661fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:28.493 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f24661fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246010afb0 con 0x7f2460070430 2023-11-15T05:01:28.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:28.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47390/0 (socket says 172.21.15.86:47390) 2023-11-15T05:01:28.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f245f7fe700 1 -- 172.21.15.86:0/2774198342 learned_addr learned my addr 172.21.15.86:0/2774198342 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:28.494 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f245f7fe700 1 -- 172.21.15.86:0/2774198342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245000b200 con 0x7f2460070430 2023-11-15T05:01:28.495 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f245f7fe700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2450002f40 tx=0x7f2450006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:28.495 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2450011690 con 0x7f2460070430 2023-11-15T05:01:28.495 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246010b1b0 con 0x7f2460070430 2023-11-15T05:01:28.495 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f245001b880 con 0x7f2460070430 2023-11-15T05:01:28.495 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f245001a6b0 con 0x7f2460070430 2023-11-15T05:01:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.486+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246010b5c0 con 0x7f2460070430 2023-11-15T05:01:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.487+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f245000f070 con 0x7f2460070430 2023-11-15T05:01:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.487+0000 7f2447fff700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 0x7f244c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.487+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2450041430 con 0x7f2460070430 2023-11-15T05:01:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.487+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460045ad0 con 0x7f2460070430 2023-11-15T05:01:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.487+0000 7f245effd700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 0x7f244c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.488+0000 7f245effd700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 0x7f244c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.492+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2450016070 con 0x7f2460070430 2023-11-15T05:01:28.643 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.640+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2460104890 con 0x7f2460070430 2023-11-15T05:01:28.643 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.641+0000 7f2447fff700 1 -- 172.21.15.86:0/2774198342 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2450022740 con 0x7f2460070430 2023-11-15T05:01:28.644 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 8443 2023-11-15T05:01:28.645 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 msgr2=0x7f244c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.645 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 0x7f244c032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).stop 2023-11-15T05:01:28.646 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 msgr2=0x7f246010aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:28.646 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2450002f40 tx=0x7f2450006ee0).stop 2023-11-15T05:01:28.646 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 shutdown_connections 2023-11-15T05:01:28.646 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f244c02feb0 0x7f244c032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.647 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 --2- 172.21.15.86:0/2774198342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2460070430 0x7f246010aac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:28.647 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.642+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 >> 172.21.15.86:0/2774198342 conn(0x7f24600f63c0 msgr2=0x7f24600f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:28.647 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.643+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 shutdown_connections 2023-11-15T05:01:28.647 INFO:teuthology.orchestra.run.smithi086.stderr:/usr/bin/ceph: 2023-11-15T05:01:28.643+0000 7f24661fe700 1 -- 172.21.15.86:0/2774198342 wait complete. 2023-11-15T05:01:28.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:27.585263+0000 mgr.smithi086.hkfamo (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:28.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:28.642376+0000 mon.smithi086 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.86:0/2774198342' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T05:01:29.020 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:01:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: disabled 2023-11-15T05:01:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:01:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:systemctl: inactive 2023-11-15T05:01:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:firewalld.service is not enabled 2023-11-15T05:01:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T05:01:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:Ceph Dashboard is now available at: 2023-11-15T05:01:29.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.040 INFO:teuthology.orchestra.run.smithi086.stderr: URL: https://smithi086.front.sepia.ceph.com:8443/ 2023-11-15T05:01:29.040 INFO:teuthology.orchestra.run.smithi086.stderr: User: admin 2023-11-15T05:01:29.041 INFO:teuthology.orchestra.run.smithi086.stderr: Password: 2gg8js3wpk 2023-11-15T05:01:29.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.042 INFO:teuthology.orchestra.run.smithi086.stderr:You can access the Ceph CLI with: 2023-11-15T05:01:29.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.042 INFO:teuthology.orchestra.run.smithi086.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:01:29.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.042 INFO:teuthology.orchestra.run.smithi086.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T05:01:29.043 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.043 INFO:teuthology.orchestra.run.smithi086.stderr: ceph telemetry on 2023-11-15T05:01:29.043 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.043 INFO:teuthology.orchestra.run.smithi086.stderr:For more information see: 2023-11-15T05:01:29.043 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.044 INFO:teuthology.orchestra.run.smithi086.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T05:01:29.044 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:01:29.044 INFO:teuthology.orchestra.run.smithi086.stderr:Bootstrap complete. 2023-11-15T05:01:29.044 INFO:teuthology.orchestra.run.smithi086.stderr:Releasing lock 140399640463848 on /run/cephadm/d8947496-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:01:29.044 INFO:teuthology.orchestra.run.smithi086.stderr:Lock 140399640463848 released on /run/cephadm/d8947496-8373-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:01:29.073 INFO:tasks.cephadm:Fetching config... 2023-11-15T05:01:29.073 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:01:29.074 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T05:01:29.131 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T05:01:29.131 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:01:29.132 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T05:01:29.188 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T05:01:29.188 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:01:29.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/mon.smithi086/keyring of=/dev/stdout 2023-11-15T05:01:29.254 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T05:01:29.255 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-11-15T05:01:29.255 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T05:01:29.312 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T05:01:29.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrDij/hiR7eSH2Qx9862ufcWnXcZFHDbQK7V5eOuqjACN8uqNohFy8aCupjmHOvbnHOmvXU5a3xOuJCAF/YydfMuml7ijTtGia+x/QrrnY2ghivPGRSxCK0QX+PfeKSBhH3ogwoc2Rz4AITt7fbG7ZI0cnFxBQrtHsaelxsS+W68prLz2dfex1YBS3DvDlcMFWAMAwe970XcfyKQ5dsU/WJ7xvWkSlVlIGSrx6XiqPyxCdqZBPL2t34WHzGE3pCQZ8xNYSIpGQc+IBCkB3GB2dDX6Y9z/1OK1ezIimcd7TeSZHmurqPSWn49XQ0K4XG5/t4goPM5pcl3SfVOjldaepvUIJO+QP2sVnr3EDYLKBmKrJvASrJq11yk3v2xvR+Lki3m0u6a6dBN3yYrS883/dF7uEC8uhIZcpYpnDE6+e6iQXF8J5+zxgvUVOe1iRBvBspErgA3sXCEu8Qra8StOv/CpiX/P01nSzEvJsTqR/c8UMOToTz5L6y5Br3AeO/lk= ceph-d8947496-8373-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T05:01:29.388 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrDij/hiR7eSH2Qx9862ufcWnXcZFHDbQK7V5eOuqjACN8uqNohFy8aCupjmHOvbnHOmvXU5a3xOuJCAF/YydfMuml7ijTtGia+x/QrrnY2ghivPGRSxCK0QX+PfeKSBhH3ogwoc2Rz4AITt7fbG7ZI0cnFxBQrtHsaelxsS+W68prLz2dfex1YBS3DvDlcMFWAMAwe970XcfyKQ5dsU/WJ7xvWkSlVlIGSrx6XiqPyxCdqZBPL2t34WHzGE3pCQZ8xNYSIpGQc+IBCkB3GB2dDX6Y9z/1OK1ezIimcd7TeSZHmurqPSWn49XQ0K4XG5/t4goPM5pcl3SfVOjldaepvUIJO+QP2sVnr3EDYLKBmKrJvASrJq11yk3v2xvR+Lki3m0u6a6dBN3yYrS883/dF7uEC8uhIZcpYpnDE6+e6iQXF8J5+zxgvUVOe1iRBvBspErgA3sXCEu8Qra8StOv/CpiX/P01nSzEvJsTqR/c8UMOToTz5L6y5Br3AeO/lk= ceph-d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:01:29.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrDij/hiR7eSH2Qx9862ufcWnXcZFHDbQK7V5eOuqjACN8uqNohFy8aCupjmHOvbnHOmvXU5a3xOuJCAF/YydfMuml7ijTtGia+x/QrrnY2ghivPGRSxCK0QX+PfeKSBhH3ogwoc2Rz4AITt7fbG7ZI0cnFxBQrtHsaelxsS+W68prLz2dfex1YBS3DvDlcMFWAMAwe970XcfyKQ5dsU/WJ7xvWkSlVlIGSrx6XiqPyxCdqZBPL2t34WHzGE3pCQZ8xNYSIpGQc+IBCkB3GB2dDX6Y9z/1OK1ezIimcd7TeSZHmurqPSWn49XQ0K4XG5/t4goPM5pcl3SfVOjldaepvUIJO+QP2sVnr3EDYLKBmKrJvASrJq11yk3v2xvR+Lki3m0u6a6dBN3yYrS883/dF7uEC8uhIZcpYpnDE6+e6iQXF8J5+zxgvUVOe1iRBvBspErgA3sXCEu8Qra8StOv/CpiX/P01nSzEvJsTqR/c8UMOToTz5L6y5Br3AeO/lk= ceph-d8947496-8373-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T05:01:29.447 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrDij/hiR7eSH2Qx9862ufcWnXcZFHDbQK7V5eOuqjACN8uqNohFy8aCupjmHOvbnHOmvXU5a3xOuJCAF/YydfMuml7ijTtGia+x/QrrnY2ghivPGRSxCK0QX+PfeKSBhH3ogwoc2Rz4AITt7fbG7ZI0cnFxBQrtHsaelxsS+W68prLz2dfex1YBS3DvDlcMFWAMAwe970XcfyKQ5dsU/WJ7xvWkSlVlIGSrx6XiqPyxCdqZBPL2t34WHzGE3pCQZ8xNYSIpGQc+IBCkB3GB2dDX6Y9z/1OK1ezIimcd7TeSZHmurqPSWn49XQ0K4XG5/t4goPM5pcl3SfVOjldaepvUIJO+QP2sVnr3EDYLKBmKrJvASrJq11yk3v2xvR+Lki3m0u6a6dBN3yYrS883/dF7uEC8uhIZcpYpnDE6+e6iQXF8J5+zxgvUVOe1iRBvBspErgA3sXCEu8Qra8StOv/CpiX/P01nSzEvJsTqR/c8UMOToTz5L6y5Br3AeO/lk= ceph-d8947496-8373-11ee-8dbe-212e2dc638e7 2023-11-15T05:01:29.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T05:01:30.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 -- 172.21.15.86:0/610452437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 msgr2=0x7f17bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:30.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/610452437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc0f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f17ac009fb0 tx=0x7f17ac009b30).stop 2023-11-15T05:01:30.640 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 -- 172.21.15.86:0/610452437 shutdown_connections 2023-11-15T05:01:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/610452437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc0f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 -- 172.21.15.86:0/610452437 >> 172.21.15.86:0/610452437 conn(0x7f17bc0f09a0 msgr2=0x7f17bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.636+0000 7f17c2f1b700 1 -- 172.21.15.86:0/610452437 shutdown_connections 2023-11-15T05:01:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c2f1b700 1 -- 172.21.15.86:0/610452437 wait complete. 2023-11-15T05:01:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c2f1b700 1 Processor -- start 2023-11-15T05:01:30.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c2f1b700 1 -- start start 2023-11-15T05:01:30.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c2f1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:30.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c2f1b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17bc060240 con 0x7f17bc0f51a0 2023-11-15T05:01:30.642 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c1f19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:30.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c1f19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47396/0 (socket says 172.21.15.86:47396) 2023-11-15T05:01:30.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.637+0000 7f17c1f19700 1 -- 172.21.15.86:0/1740527534 learned_addr learned my addr 172.21.15.86:0/1740527534 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:30.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17c1f19700 1 -- 172.21.15.86:0/1740527534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac005040 con 0x7f17bc0f51a0 2023-11-15T05:01:30.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17c1f19700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f17ac002040 tx=0x7f17ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:30.643 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ac0095f0 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f17ac007e30 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17bc060440 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ac014d20 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17bc109840 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.638+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f17ac025070 con 0x7f17bc0f51a0 2023-11-15T05:01:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.639+0000 7f17b27fc700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 0x7f17a80323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.639+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17ac040920 con 0x7f17bc0f51a0 2023-11-15T05:01:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.639+0000 7f17c1718700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 0x7f17a80323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.639+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17bc059100 con 0x7f17bc0f51a0 2023-11-15T05:01:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.639+0000 7f17c1718700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 0x7f17a80323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.643+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17ac00f300 con 0x7f17bc0f51a0 2023-11-15T05:01:30.795 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.794+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f17bc109ff0 con 0x7f17bc0f51a0 2023-11-15T05:01:30.798 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.797+0000 7f17b27fc700 1 -- 172.21.15.86:0/1740527534 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f17ac02d0c0 con 0x7f17bc0f51a0 2023-11-15T05:01:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 msgr2=0x7f17a80323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 0x7f17a80323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f17b80096a0 tx=0x7f17b8006b40).stop 2023-11-15T05:01:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 msgr2=0x7f17bc109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f17ac002040 tx=0x7f17ac00b040).stop 2023-11-15T05:01:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 shutdown_connections 2023-11-15T05:01:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f17a802ff00 0x7f17a80323c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 --2- 172.21.15.86:0/1740527534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17bc0f51a0 0x7f17bc109190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 >> 172.21.15.86:0/1740527534 conn(0x7f17bc0f09a0 msgr2=0x7f17bc0f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:30.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 shutdown_connections 2023-11-15T05:01:30.802 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:30.799+0000 7f17c2f1b700 1 -- 172.21.15.86:0/1740527534 wait complete. 2023-11-15T05:01:31.164 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2023-11-15T05:01:31.164 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:01:31.164 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2023-11-15T05:01:31.186 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2023-11-15T05:01:31.186 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T05:01:31.247 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2023-11-15T05:01:31.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi154 2023-11-15T05:01:32.125 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:30.798691+0000 mon.smithi086 (mon.0) 107 : audit [INF] from='client.? 172.21.15.86:0/1740527534' entity='client.admin' 2023-11-15T05:01:32.126 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:30.806885+0000 mon.smithi086 (mon.0) 108 : cluster [DBG] mgrmap e16: smithi086.hkfamo(active, since 6s) 2023-11-15T05:01:33.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:32.733009+0000 mon.smithi086 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:33.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:32.948599+0000 mon.smithi086 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:33.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:33.524216+0000 mon.smithi086 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:37.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:35.763166+0000 mon.smithi086 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:37.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:36.029235+0000 mon.smithi086 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:37.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:36 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:36.031305+0000 mon.smithi086 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:38.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:36.037094+0000 mgr.smithi086.hkfamo (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi086 on smithi086 2023-11-15T05:01:46.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:44.797064+0000 mgr.smithi086.hkfamo (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:46.797507+0000 mgr.smithi086.hkfamo (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:49.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:48.797888+0000 mgr.smithi086.hkfamo (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:51.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:50.798137+0000 mgr.smithi086.hkfamo (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:53.048 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.046+0000 7f8cdac22700 1 -- 172.21.15.154:0/1219143972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd40f6510 msgr2=0x7f8cd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.046+0000 7f8cdac22700 1 --2- 172.21.15.154:0/1219143972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd40f6510 0x7f8cd40f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8cc8009fb0 tx=0x7f8cc8009b30).stop 2023-11-15T05:01:53.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- 172.21.15.154:0/1219143972 shutdown_connections 2023-11-15T05:01:53.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 --2- 172.21.15.154:0/1219143972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd40f6510 0x7f8cd40f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- 172.21.15.154:0/1219143972 >> 172.21.15.154:0/1219143972 conn(0x7f8cd40f1c70 msgr2=0x7f8cd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:53.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- 172.21.15.154:0/1219143972 shutdown_connections 2023-11-15T05:01:53.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- 172.21.15.154:0/1219143972 wait complete. 2023-11-15T05:01:53.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 Processor -- start 2023-11-15T05:01:53.553 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:52.798512+0000 mgr.smithi086.hkfamo (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:53.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- start start 2023-11-15T05:01:53.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:53.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.047+0000 7f8cdac22700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4107140 con 0x7f8cd4106830 2023-11-15T05:01:53.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd89be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:53.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd89be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39658/0 (socket says 172.21.15.154:39658) 2023-11-15T05:01:53.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd89be700 1 -- 172.21.15.154:0/545422040 learned_addr learned my addr 172.21.15.154:0/545422040 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:01:53.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd89be700 1 -- 172.21.15.154:0/545422040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8005040 con 0x7f8cd4106830 2023-11-15T05:01:53.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd89be700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8cc8009f80 tx=0x7f8cc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:53.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc80136c0 con 0x7f8cd4106830 2023-11-15T05:01:53.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd4107340 con 0x7f8cd4106830 2023-11-15T05:01:53.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc8013e70 con 0x7f8cd4106830 2023-11-15T05:01:53.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc8014420 con 0x7f8cd4106830 2023-11-15T05:01:53.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.048+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd4107e90 con 0x7f8cd4106830 2023-11-15T05:01:53.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.049+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc8025060 con 0x7f8cd4106830 2023-11-15T05:01:53.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.049+0000 7f8cd17fa700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 0x7f8cc4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:53.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.049+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd4059100 con 0x7f8cd4106830 2023-11-15T05:01:53.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.049+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc8040f10 con 0x7f8cd4106830 2023-11-15T05:01:53.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.050+0000 7f8cd3fff700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 0x7f8cc4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:53.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.050+0000 7f8cd3fff700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 0x7f8cc4032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:53.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.054+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc801cb30 con 0x7f8cd4106830 2023-11-15T05:01:53.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.223+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cd4108670 con 0x7f8cc402ffb0 2023-11-15T05:01:53.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.822+0000 7f8cd17fa700 1 -- 172.21.15.154:0/545422040 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f8cd4108670 con 0x7f8cc402ffb0 2023-11-15T05:01:53.824 INFO:teuthology.orchestra.run.smithi154.stdout:Added host 'smithi154' 2023-11-15T05:01:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 msgr2=0x7f8cc4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 0x7f8cc4032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).stop 2023-11-15T05:01:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 msgr2=0x7f8cd4106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8cc8009f80 tx=0x7f8cc8009b30).stop 2023-11-15T05:01:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 shutdown_connections 2023-11-15T05:01:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8cc402ffb0 0x7f8cc4032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 --2- 172.21.15.154:0/545422040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cd4106830 0x7f8cd4106c50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 >> 172.21.15.154:0/545422040 conn(0x7f8cd40f1c70 msgr2=0x7f8cd40f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 shutdown_connections 2023-11-15T05:01:53.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:53.824+0000 7f8cdac22700 1 -- 172.21.15.154:0/545422040 wait complete. 2023-11-15T05:01:54.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T05:01:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:53.225590+0000 mgr.smithi086.hkfamo (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:01:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:53.821919+0000 mon.smithi086 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:01:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:53.822313+0000 mgr.smithi086.hkfamo (mgr.14162) 12 : cephadm [INF] Added host smithi154 2023-11-15T05:01:55.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:54.798815+0000 mgr.smithi086.hkfamo (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:55.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- 172.21.15.154:0/3723189241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 msgr2=0x7fc5d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:55.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 --2- 172.21.15.154:0/3723189241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d00f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc5c4009fb0 tx=0x7fc5c4009b30).stop 2023-11-15T05:01:55.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- 172.21.15.154:0/3723189241 shutdown_connections 2023-11-15T05:01:55.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 --2- 172.21.15.154:0/3723189241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d00f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:55.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- 172.21.15.154:0/3723189241 >> 172.21.15.154:0/3723189241 conn(0x7fc5d00f1c70 msgr2=0x7fc5d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:55.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- 172.21.15.154:0/3723189241 shutdown_connections 2023-11-15T05:01:55.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- 172.21.15.154:0/3723189241 wait complete. 2023-11-15T05:01:55.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 Processor -- start 2023-11-15T05:01:55.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.254+0000 7fc5d7234700 1 -- start start 2023-11-15T05:01:55.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d7234700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:55.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d7234700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0069e20 con 0x7fc5d00f6510 2023-11-15T05:01:55.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d4fd0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:55.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d4fd0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59078/0 (socket says 172.21.15.154:59078) 2023-11-15T05:01:55.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d4fd0700 1 -- 172.21.15.154:0/1658008247 learned_addr learned my addr 172.21.15.154:0/1658008247 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:01:55.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d4fd0700 1 -- 172.21.15.154:0/1658008247 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4005040 con 0x7fc5d00f6510 2023-11-15T05:01:55.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5d4fd0700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc5c4013510 tx=0x7fc5c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:55.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.255+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c4009320 con 0x7fc5d00f6510 2023-11-15T05:01:55.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5c4013f00 con 0x7fc5d00f6510 2023-11-15T05:01:55.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0069fc0 con 0x7fc5d00f6510 2023-11-15T05:01:55.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c4014d90 con 0x7fc5d00f6510 2023-11-15T05:01:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d006a450 con 0x7fc5d00f6510 2023-11-15T05:01:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5c4014420 con 0x7fc5d00f6510 2023-11-15T05:01:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.256+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d0059100 con 0x7fc5d00f6510 2023-11-15T05:01:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.257+0000 7fc5cd7fa700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.257+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5c4041080 con 0x7fc5d00f6510 2023-11-15T05:01:55.262 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.257+0000 7fc5cffff700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:55.262 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.257+0000 7fc5cffff700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 0x7fc5c0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc5bc0096a0 tx=0x7fc5bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:55.262 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.261+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5c40095f0 con 0x7fc5d00f6510 2023-11-15T05:01:55.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.431+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5d006aca0 con 0x7fc5c002ff60 2023-11-15T05:01:55.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.433+0000 7fc5cd7fa700 1 -- 172.21.15.154:0/1658008247 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fc5d006aca0 con 0x7fc5c002ff60 2023-11-15T05:01:55.434 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:01:55.434 INFO:teuthology.orchestra.run.smithi154.stdout:[{"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "smithi154", "hostname": "smithi154", "labels": [], "status": ""}] 2023-11-15T05:01:55.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.434+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 msgr2=0x7fc5c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:55.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.434+0000 7fc5d7234700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 0x7fc5c0032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc5bc0096a0 tx=0x7fc5bc006b40).stop 2023-11-15T05:01:55.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.434+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 msgr2=0x7fc5d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:55.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc5c4013510 tx=0x7fc5c400b040).stop 2023-11-15T05:01:55.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 shutdown_connections 2023-11-15T05:01:55.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc5c002ff60 0x7fc5c0032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:55.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 --2- 172.21.15.154:0/1658008247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5d00f6510 0x7fc5d006b9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:55.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 >> 172.21.15.154:0/1658008247 conn(0x7fc5d00f1c70 msgr2=0x7fc5d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:55.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 shutdown_connections 2023-11-15T05:01:55.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:55.435+0000 7fc5d7234700 1 -- 172.21.15.154:0/1658008247 wait complete. 2023-11-15T05:01:55.799 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T05:01:55.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T05:01:56.106 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:55 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:55.433554+0000 mgr.smithi086.hkfamo (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T05:01:56.947 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- 172.21.15.86:0/476138142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 msgr2=0x7f39840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:56.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 --2- 172.21.15.86:0/476138142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f39840f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3978009fb0 tx=0x7f3978009b30).stop 2023-11-15T05:01:56.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- 172.21.15.86:0/476138142 shutdown_connections 2023-11-15T05:01:56.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 --2- 172.21.15.86:0/476138142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f39840f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:56.948 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- 172.21.15.86:0/476138142 >> 172.21.15.86:0/476138142 conn(0x7f39840f0b70 msgr2=0x7f39840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:56.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- 172.21.15.86:0/476138142 shutdown_connections 2023-11-15T05:01:56.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- 172.21.15.86:0/476138142 wait complete. 2023-11-15T05:01:56.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 Processor -- start 2023-11-15T05:01:56.949 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 -- start start 2023-11-15T05:01:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.946+0000 7f3988d25700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3988d25700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3984106c20 con 0x7f39840f5410 2023-11-15T05:01:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f398259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f398259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40922/0 (socket says 172.21.15.86:40922) 2023-11-15T05:01:56.950 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f398259c700 1 -- 172.21.15.86:0/1471074272 learned_addr learned my addr 172.21.15.86:0/1471074272 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-11-15T05:01:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f398259c700 1 -- 172.21.15.86:0/1471074272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978005040 con 0x7f39840f5410 2023-11-15T05:01:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f398259c700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3978013510 tx=0x7f397800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3978009320 con 0x7f39840f5410 2023-11-15T05:01:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3984106e20 con 0x7f39840f5410 2023-11-15T05:01:56.951 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3978013f00 con 0x7f39840f5410 2023-11-15T05:01:56.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3978014d90 con 0x7f39840f5410 2023-11-15T05:01:56.952 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.947+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39841072b0 con 0x7f39840f5410 2023-11-15T05:01:56.953 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.948+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39780095f0 con 0x7f39840f5410 2023-11-15T05:01:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.948+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3984045ad0 con 0x7f39840f5410 2023-11-15T05:01:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.948+0000 7f3972ffd700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 0x7f396c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.948+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3978040fc0 con 0x7f39840f5410 2023-11-15T05:01:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.948+0000 7f3981d9b700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 0x7f396c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.949+0000 7f3981d9b700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 0x7f396c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f39740096a0 tx=0x7f3974006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:56.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:56.952+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3978014420 con 0x7f39840f5410 2023-11-15T05:01:57.107 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.106+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f39840fae60 con 0x7f39840f5410 2023-11-15T05:01:57.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.283+0000 7f3972ffd700 1 -- 172.21.15.86:0/1471074272 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3978010230 con 0x7f39840f5410 2023-11-15T05:01:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 msgr2=0x7f396c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 0x7f396c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f39740096a0 tx=0x7f3974006b40).stop 2023-11-15T05:01:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 msgr2=0x7f3984106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3978013510 tx=0x7f397800b040).stop 2023-11-15T05:01:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 shutdown_connections 2023-11-15T05:01:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f396c02ff60 0x7f396c032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 --2- 172.21.15.86:0/1471074272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39840f5410 0x7f3984106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:57.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 >> 172.21.15.86:0/1471074272 conn(0x7f39840f0b70 msgr2=0x7f39840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:57.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 shutdown_connections 2023-11-15T05:01:57.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-11-15T05:01:57.285+0000 7f3988d25700 1 -- 172.21.15.86:0/1471074272 wait complete. 2023-11-15T05:01:57.288 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2023-11-15T05:01:57.593 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:56.799073+0000 mgr.smithi086.hkfamo (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:57.594 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:57.107604+0000 mon.smithi086 (mon.0) 116 : audit [INF] from='client.? 172.21.15.86:0/1471074272' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T05:01:57.682 INFO:tasks.cephadm:Adding mon.smithi086 on smithi086 2023-11-15T05:01:57.683 INFO:tasks.cephadm:Adding mon.smithi154 on smithi154 2023-11-15T05:01:57.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154' 2023-11-15T05:01:58.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:57.283784+0000 mon.smithi086 (mon.0) 117 : audit [INF] from='client.? 172.21.15.86:0/1471074272' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T05:01:58.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:57.283832+0000 mon.smithi086 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T05:01:58.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- 172.21.15.154:0/1407108273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 msgr2=0x7f320c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:58.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 --2- 172.21.15.154:0/1407108273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c0f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f31fc009fb0 tx=0x7f31fc009b30).stop 2023-11-15T05:01:58.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- 172.21.15.154:0/1407108273 shutdown_connections 2023-11-15T05:01:58.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 --2- 172.21.15.154:0/1407108273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c0f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:58.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- 172.21.15.154:0/1407108273 >> 172.21.15.154:0/1407108273 conn(0x7f320c0f0b70 msgr2=0x7f320c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:58.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- 172.21.15.154:0/1407108273 shutdown_connections 2023-11-15T05:01:58.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- 172.21.15.154:0/1407108273 wait complete. 2023-11-15T05:01:58.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 Processor -- start 2023-11-15T05:01:58.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.819+0000 7f3211dae700 1 -- start start 2023-11-15T05:01:58.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f3211dae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:58.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f3211dae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c060220 con 0x7f320c0f5410 2023-11-15T05:01:58.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f320b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:58.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f320b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59104/0 (socket says 172.21.15.154:59104) 2023-11-15T05:01:58.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f320b7fe700 1 -- 172.21.15.154:0/3161635238 learned_addr learned my addr 172.21.15.154:0/3161635238 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:01:58.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f320b7fe700 1 -- 172.21.15.154:0/3161635238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc005040 con 0x7f320c0f5410 2023-11-15T05:01:58.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f320b7fe700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f31fc013510 tx=0x7f31fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:58.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31fc009320 con 0x7f320c0f5410 2023-11-15T05:01:58.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c060420 con 0x7f320c0f5410 2023-11-15T05:01:58.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31fc013f00 con 0x7f320c0f5410 2023-11-15T05:01:58.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31fc014d90 con 0x7f320c0f5410 2023-11-15T05:01:58.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.820+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c109b00 con 0x7f320c0f5410 2023-11-15T05:01:58.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.821+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31fc014420 con 0x7f320c0f5410 2023-11-15T05:01:58.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.821+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c059100 con 0x7f320c0f5410 2023-11-15T05:01:58.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.822+0000 7f31f3fff700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 0x7f31f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:01:58.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.822+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31fc040fb0 con 0x7f320c0f5410 2023-11-15T05:01:58.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.822+0000 7f320affd700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 0x7f31f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:01:58.829 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.822+0000 7f320affd700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 0x7f31f4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f32000096a0 tx=0x7f3200006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:01:58.829 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:58.826+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31fc0095f0 con 0x7f320c0f5410 2023-11-15T05:01:59.005 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.004+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 --> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154", "target": ["mon-mgr", ""]}) v1 -- 0x7f320c0f1e00 con 0x7f31f402ff60 2023-11-15T05:01:59.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.010+0000 7f31f3fff700 1 -- 172.21.15.154:0/3161635238 <== mgr.14162 v2:172.21.15.86:6800/432331256 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f320c0f1e00 con 0x7f31f402ff60 2023-11-15T05:01:59.011 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mon update... 2023-11-15T05:01:59.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 msgr2=0x7f31f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:59.012 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 0x7f31f4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f32000096a0 tx=0x7f3200006b40).stop 2023-11-15T05:01:59.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 msgr2=0x7f320c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:01:59.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f31fc013510 tx=0x7f31fc00b040).stop 2023-11-15T05:01:59.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 shutdown_connections 2023-11-15T05:01:59.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.011+0000 7f3211dae700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f31f402ff60 0x7f31f4032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:59.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.012+0000 7f3211dae700 1 --2- 172.21.15.154:0/3161635238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f320c0f5410 0x7f320c109450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:01:59.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.012+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 >> 172.21.15.154:0/3161635238 conn(0x7f320c0f0b70 msgr2=0x7f320c0f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:01:59.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.012+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 shutdown_connections 2023-11-15T05:01:59.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:01:59.012+0000 7f3211dae700 1 -- 172.21.15.154:0/3161635238 wait complete. 2023-11-15T05:01:59.386 DEBUG:teuthology.orchestra.run.smithi154:mon.smithi154> sudo journalctl -f -n 0 -u ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154.service 2023-11-15T05:01:59.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:01:59.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:01:59.417 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:-- Logs begin at Wed 2023-11-15 04:48:34 UTC. -- 2023-11-15T05:01:59.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:01:58.799433+0000 mgr.smithi086.hkfamo (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:01:59.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:01:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:59.010733+0000 mon.smithi086 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:02:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/2804584960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 msgr2=0x7f3c580f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/2804584960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f9190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3c48009fb0 tx=0x7f3c48009b30).stop 2023-11-15T05:02:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/2804584960 shutdown_connections 2023-11-15T05:02:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/2804584960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f9190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/2804584960 >> 172.21.15.154:0/2804584960 conn(0x7f3c580f0b70 msgr2=0x7f3c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/2804584960 shutdown_connections 2023-11-15T05:02:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.483+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/2804584960 wait complete. 2023-11-15T05:02:00.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c5e6f0700 1 Processor -- start 2023-11-15T05:02:00.487 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c5e6f0700 1 -- start start 2023-11-15T05:02:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c5e6f0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c5e6f0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580f5090 con 0x7f3c580601e0 2023-11-15T05:02:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59110/0 (socket says 172.21.15.154:59110) 2023-11-15T05:02:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c57fff700 1 -- 172.21.15.154:0/3748797499 learned_addr learned my addr 172.21.15.154:0/3748797499 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.484+0000 7f3c57fff700 1 -- 172.21.15.154:0/3748797499 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48005040 con 0x7f3c580601e0 2023-11-15T05:02:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c57fff700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c48004dd0 tx=0x7f3c48009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c48018070 con 0x7f3c580601e0 2023-11-15T05:02:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c580f5230 con 0x7f3c580601e0 2023-11-15T05:02:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c48013410 con 0x7f3c580601e0 2023-11-15T05:02:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c4801c3c0 con 0x7f3c580601e0 2023-11-15T05:02:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.485+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c580f56c0 con 0x7f3c580601e0 2023-11-15T05:02:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c4801d420 con 0x7f3c580601e0 2023-11-15T05:02:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58059100 con 0x7f3c580601e0 2023-11-15T05:02:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c54ff9700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:00.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c480413d0 con 0x7f3c580601e0 2023-11-15T05:02:00.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c577fe700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:00.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.486+0000 7f3c577fe700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 0x7f3c40032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3c4c0096a0 tx=0x7f3c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:00.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.490+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c4801d820 con 0x7f3c580601e0 2023-11-15T05:02:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.696+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c580fbf60 con 0x7f3c580601e0 2023-11-15T05:02:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.697+0000 7f3c54ff9700 1 -- 172.21.15.154:0/3748797499 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c4801b370 con 0x7f3c580601e0 2023-11-15T05:02:00.698 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:00.698 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:00.699 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 msgr2=0x7f3c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:00.699 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 0x7f3c40032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3c4c0096a0 tx=0x7f3c4c006b40).stop 2023-11-15T05:02:00.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 msgr2=0x7f3c580f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:00.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3c48004dd0 tx=0x7f3c48009b30).stop 2023-11-15T05:02:00.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 shutdown_connections 2023-11-15T05:02:00.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:00.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 --2- 172.21.15.154:0/3748797499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c580601e0 0x7f3c580f6c10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:00.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.698+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 >> 172.21.15.154:0/3748797499 conn(0x7f3c580f0b70 msgr2=0x7f3c580f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:00.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.699+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 shutdown_connections 2023-11-15T05:02:00.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:00.699+0000 7f3c5e6f0700 1 -- 172.21.15.154:0/3748797499 wait complete. 2023-11-15T05:02:00.702 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:00.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:01:59.006686+0000 mgr.smithi086.hkfamo (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:02:00.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:01:59.008037+0000 mgr.smithi086.hkfamo (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi086:172.21.15.86=smithi086;smithi154:172.21.15.154=smithi154;count:2 2023-11-15T05:02:01.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:01 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:00.697692+0000 mon.smithi086 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.154:0/3748797499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:01.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:01 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:00.799720+0000 mgr.smithi086.hkfamo (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:02.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:02.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:03.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3616809649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 msgr2=0x7f7e580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:03.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3616809649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e580f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7e48009fb0 tx=0x7f7e48009b30).stop 2023-11-15T05:02:03.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3616809649 shutdown_connections 2023-11-15T05:02:03.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3616809649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e580f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:03.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3616809649 >> 172.21.15.154:0/3616809649 conn(0x7f7e580f0b70 msgr2=0x7f7e580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:03.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3616809649 shutdown_connections 2023-11-15T05:02:03.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.173+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3616809649 wait complete. 2023-11-15T05:02:03.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5f43c700 1 Processor -- start 2023-11-15T05:02:03.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5f43c700 1 -- start start 2023-11-15T05:02:03.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5f43c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:03.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5f43c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58108e10 con 0x7f7e580f5410 2023-11-15T05:02:03.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5d1d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:03.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5d1d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59118/0 (socket says 172.21.15.154:59118) 2023-11-15T05:02:03.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.174+0000 7f7e5d1d8700 1 -- 172.21.15.154:0/3901269569 learned_addr learned my addr 172.21.15.154:0/3901269569 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:03.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e5d1d8700 1 -- 172.21.15.154:0/3901269569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48005040 con 0x7f7e580f5410 2023-11-15T05:02:03.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e5d1d8700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7e48013510 tx=0x7f7e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:03.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e48009320 con 0x7f7e580f5410 2023-11-15T05:02:03.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e58109010 con 0x7f7e580f5410 2023-11-15T05:02:03.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e48013f00 con 0x7f7e580f5410 2023-11-15T05:02:03.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e48014d90 con 0x7f7e580f5410 2023-11-15T05:02:03.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.175+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e581094a0 con 0x7f7e580f5410 2023-11-15T05:02:03.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.176+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e48014420 con 0x7f7e580f5410 2023-11-15T05:02:03.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.176+0000 7f7e4dffb700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 0x7f7e44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:03.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.176+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e58045ad0 con 0x7f7e580f5410 2023-11-15T05:02:03.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.176+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e48041110 con 0x7f7e580f5410 2023-11-15T05:02:03.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.176+0000 7f7e5c9d7700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 0x7f7e44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:03.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.177+0000 7f7e5c9d7700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 0x7f7e44032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e540096a0 tx=0x7f7e54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:03.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.181+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e48012330 con 0x7f7e580f5410 2023-11-15T05:02:03.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:02.800092+0000 mgr.smithi086.hkfamo (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:03.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.389+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e5810a350 con 0x7f7e580f5410 2023-11-15T05:02:03.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.389+0000 7f7e4dffb700 1 -- 172.21.15.154:0/3901269569 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e480221c0 con 0x7f7e580f5410 2023-11-15T05:02:03.391 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:03.391 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:03.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 msgr2=0x7f7e44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:03.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 0x7f7e44032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7e540096a0 tx=0x7f7e54006b40).stop 2023-11-15T05:02:03.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 msgr2=0x7f7e58108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:03.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7e48013510 tx=0x7f7e4800b040).stop 2023-11-15T05:02:03.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 shutdown_connections 2023-11-15T05:02:03.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e4402ff60 0x7f7e44032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:03.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 --2- 172.21.15.154:0/3901269569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e580f5410 0x7f7e58108920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:03.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 >> 172.21.15.154:0/3901269569 conn(0x7f7e580f0b70 msgr2=0x7f7e5805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:03.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 shutdown_connections 2023-11-15T05:02:03.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:03.391+0000 7f7e5f43c700 1 -- 172.21.15.154:0/3901269569 wait complete. 2023-11-15T05:02:03.394 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:04.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:03.390201+0000 mon.smithi086 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.154:0/3901269569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:04.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:04.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:04.800389+0000 mgr.smithi086.hkfamo (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:05.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 -- 172.21.15.154:0/2115628099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 msgr2=0x7f30380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:05.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 --2- 172.21.15.154:0/2115628099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3020009fb0 tx=0x7f3020009b30).stop 2023-11-15T05:02:05.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 -- 172.21.15.154:0/2115628099 shutdown_connections 2023-11-15T05:02:05.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 --2- 172.21.15.154:0/2115628099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:05.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 -- 172.21.15.154:0/2115628099 >> 172.21.15.154:0/2115628099 conn(0x7f30380f0b70 msgr2=0x7f30380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:05.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 -- 172.21.15.154:0/2115628099 shutdown_connections 2023-11-15T05:02:05.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.919+0000 7f303d999700 1 -- 172.21.15.154:0/2115628099 wait complete. 2023-11-15T05:02:05.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f303d999700 1 Processor -- start 2023-11-15T05:02:05.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f303d999700 1 -- start start 2023-11-15T05:02:05.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f303d999700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:05.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f303d999700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30380fb820 con 0x7f30380f5410 2023-11-15T05:02:05.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f3036ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:05.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f3036ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:52620/0 (socket says 172.21.15.154:52620) 2023-11-15T05:02:05.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.920+0000 7f3036ffd700 1 -- 172.21.15.154:0/119573667 learned_addr learned my addr 172.21.15.154:0/119573667 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:05.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f3036ffd700 1 -- 172.21.15.154:0/119573667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3020005040 con 0x7f30380f5410 2023-11-15T05:02:05.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f3036ffd700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3020014000 tx=0x7f302000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:05.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30200095f0 con 0x7f30380f5410 2023-11-15T05:02:05.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30380fb9c0 con 0x7f30380f5410 2023-11-15T05:02:05.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3020007e70 con 0x7f30380f5410 2023-11-15T05:02:05.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3020014ea0 con 0x7f30380f5410 2023-11-15T05:02:05.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.921+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30380fbe50 con 0x7f30380f5410 2023-11-15T05:02:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3038059100 con 0x7f30380f5410 2023-11-15T05:02:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3020025070 con 0x7f30380f5410 2023-11-15T05:02:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f302f7fe700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30200409f0 con 0x7f30380f5410 2023-11-15T05:02:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f30367fc700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:05.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.922+0000 7f30367fc700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 0x7f3024032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:05.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:05.926+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f302000f4e0 con 0x7f30380f5410 2023-11-15T05:02:06.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.135+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30380fae60 con 0x7f30380f5410 2023-11-15T05:02:06.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.136+0000 7f302f7fe700 1 -- 172.21.15.154:0/119573667 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f302000f300 con 0x7f30380f5410 2023-11-15T05:02:06.137 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:06.137 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:06.139 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 msgr2=0x7f3024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:06.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 0x7f3024032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f30280096a0 tx=0x7f3028006b40).stop 2023-11-15T05:02:06.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 msgr2=0x7f30380fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:06.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3020014000 tx=0x7f302000b040).stop 2023-11-15T05:02:06.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 shutdown_connections 2023-11-15T05:02:06.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f302402ff60 0x7f3024032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:06.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 --2- 172.21.15.154:0/119573667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30380f5410 0x7f30380fd3a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:06.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 >> 172.21.15.154:0/119573667 conn(0x7f30380f0b70 msgr2=0x7f303805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:06.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 shutdown_connections 2023-11-15T05:02:06.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:06.137+0000 7f303d999700 1 -- 172.21.15.154:0/119573667 wait complete. 2023-11-15T05:02:06.144 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:06.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:06.136667+0000 mon.smithi086 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.154:0/119573667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:07.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:07.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:06.800751+0000 mgr.smithi086.hkfamo (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 -- 172.21.15.154:0/3407229444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 msgr2=0x7f1a100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 --2- 172.21.15.154:0/3407229444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a100f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1a04009fb0 tx=0x7f1a04009b30).stop 2023-11-15T05:02:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 -- 172.21.15.154:0/3407229444 shutdown_connections 2023-11-15T05:02:08.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 --2- 172.21.15.154:0/3407229444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a100f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:08.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 -- 172.21.15.154:0/3407229444 >> 172.21.15.154:0/3407229444 conn(0x7f1a100f1c70 msgr2=0x7f1a100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:08.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 -- 172.21.15.154:0/3407229444 shutdown_connections 2023-11-15T05:02:08.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.661+0000 7f1a15758700 1 -- 172.21.15.154:0/3407229444 wait complete. 2023-11-15T05:02:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a15758700 1 Processor -- start 2023-11-15T05:02:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a15758700 1 -- start start 2023-11-15T05:02:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a15758700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a15758700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10106c20 con 0x7f1a100f6510 2023-11-15T05:02:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a0effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a0effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:52624/0 (socket says 172.21.15.154:52624) 2023-11-15T05:02:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a0effd700 1 -- 172.21.15.154:0/2681867180 learned_addr learned my addr 172.21.15.154:0/2681867180 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.662+0000 7f1a0effd700 1 -- 172.21.15.154:0/2681867180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04005040 con 0x7f1a100f6510 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f1a0effd700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1a04013510 tx=0x7f1a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a04009320 con 0x7f1a100f6510 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a10106e20 con 0x7f1a100f6510 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a04013f00 con 0x7f1a100f6510 2023-11-15T05:02:08.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a04014d90 con 0x7f1a100f6510 2023-11-15T05:02:08.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.663+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a101072b0 con 0x7f1a100f6510 2023-11-15T05:02:08.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.664+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10045ad0 con 0x7f1a100f6510 2023-11-15T05:02:08.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.664+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a04014420 con 0x7f1a100f6510 2023-11-15T05:02:08.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.664+0000 7f19f77fe700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 0x7f19f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:08.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.664+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a04041000 con 0x7f1a100f6510 2023-11-15T05:02:08.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.664+0000 7f1a0e7fc700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 0x7f19f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:08.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.665+0000 7f1a0e7fc700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 0x7f19f8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a000096a0 tx=0x7f1a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:08.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.669+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a040095f0 con 0x7f1a100f6510 2023-11-15T05:02:08.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.879+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a100fbf60 con 0x7f1a100f6510 2023-11-15T05:02:08.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.880+0000 7f19f77fe700 1 -- 172.21.15.154:0/2681867180 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a0401db70 con 0x7f1a100f6510 2023-11-15T05:02:08.881 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:08.881 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:08.882 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.881+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 msgr2=0x7f19f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:08.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.881+0000 7f1a15758700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 0x7f19f8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1a000096a0 tx=0x7f1a00006b40).stop 2023-11-15T05:02:08.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.881+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 msgr2=0x7f1a10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:08.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.881+0000 7f1a15758700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1a04013510 tx=0x7f1a0400b040).stop 2023-11-15T05:02:08.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 shutdown_connections 2023-11-15T05:02:08.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f19f802ff60 0x7f19f8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:08.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 --2- 172.21.15.154:0/2681867180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a100f6510 0x7f1a10106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:08.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 >> 172.21.15.154:0/2681867180 conn(0x7f1a100f1c70 msgr2=0x7f1a100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:08.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 shutdown_connections 2023-11-15T05:02:08.884 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:08.882+0000 7f1a15758700 1 -- 172.21.15.154:0/2681867180 wait complete. 2023-11-15T05:02:08.885 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:09.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:08.801031+0000 mgr.smithi086.hkfamo (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:10.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:08.880617+0000 mon.smithi086 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.154:0/2681867180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:10.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:10.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:10.801315+0000 mgr.smithi086.hkfamo (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:11.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 -- 172.21.15.154:0/1110268080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 msgr2=0x7f0c7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:11.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 --2- 172.21.15.154:0/1110268080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0c70009fb0 tx=0x7f0c70009b30).stop 2023-11-15T05:02:11.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 -- 172.21.15.154:0/1110268080 shutdown_connections 2023-11-15T05:02:11.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 --2- 172.21.15.154:0/1110268080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:11.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 -- 172.21.15.154:0/1110268080 >> 172.21.15.154:0/1110268080 conn(0x7f0c7c0f0b70 msgr2=0x7f0c7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:11.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 -- 172.21.15.154:0/1110268080 shutdown_connections 2023-11-15T05:02:11.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.391+0000 7f0c80d00700 1 -- 172.21.15.154:0/1110268080 wait complete. 2023-11-15T05:02:11.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c80d00700 1 Processor -- start 2023-11-15T05:02:11.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c80d00700 1 -- start start 2023-11-15T05:02:11.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c80d00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:11.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c80d00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c107240 con 0x7f0c7c0f5410 2023-11-15T05:02:11.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c7a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:11.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c7a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:52636/0 (socket says 172.21.15.154:52636) 2023-11-15T05:02:11.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c7a59c700 1 -- 172.21.15.154:0/2194779127 learned_addr learned my addr 172.21.15.154:0/2194779127 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:11.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.392+0000 7f0c7a59c700 1 -- 172.21.15.154:0/2194779127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70005040 con 0x7f0c7c0f5410 2023-11-15T05:02:11.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c7a59c700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0c70013510 tx=0x7f0c7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:11.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c70009320 con 0x7f0c7c0f5410 2023-11-15T05:02:11.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c060220 con 0x7f0c7c0f5410 2023-11-15T05:02:11.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c70013f00 con 0x7f0c7c0f5410 2023-11-15T05:02:11.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c70014d90 con 0x7f0c7c0f5410 2023-11-15T05:02:11.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.393+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c10bd40 con 0x7f0c7c0f5410 2023-11-15T05:02:11.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.394+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c700095f0 con 0x7f0c7c0f5410 2023-11-15T05:02:11.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.394+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c045ad0 con 0x7f0c7c0f5410 2023-11-15T05:02:11.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.394+0000 7f0c6affd700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 0x7f0c64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:11.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.394+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c70041000 con 0x7f0c7c0f5410 2023-11-15T05:02:11.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.394+0000 7f0c79d9b700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 0x7f0c64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:11.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.395+0000 7f0c79d9b700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 0x7f0c64032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0c6c0096a0 tx=0x7f0c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:11.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.399+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c70014420 con 0x7f0c7c0f5410 2023-11-15T05:02:11.608 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.607+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c7c0fae60 con 0x7f0c7c0f5410 2023-11-15T05:02:11.608 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.608+0000 7f0c6affd700 1 -- 172.21.15.154:0/2194779127 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c70010230 con 0x7f0c7c0f5410 2023-11-15T05:02:11.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:11.609 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:11.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 msgr2=0x7f0c64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:11.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 0x7f0c64032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0c6c0096a0 tx=0x7f0c6c006b40).stop 2023-11-15T05:02:11.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 msgr2=0x7f0c7c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:11.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0c70013510 tx=0x7f0c7000b040).stop 2023-11-15T05:02:11.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 shutdown_connections 2023-11-15T05:02:11.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0c6402ff60 0x7f0c64032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:11.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 --2- 172.21.15.154:0/2194779127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c7c0f5410 0x7f0c7c10b850 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:11.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 >> 172.21.15.154:0/2194779127 conn(0x7f0c7c0f0b70 msgr2=0x7f0c7c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:11.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 shutdown_connections 2023-11-15T05:02:11.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:11.609+0000 7f0c80d00700 1 -- 172.21.15.154:0/2194779127 wait complete. 2023-11-15T05:02:11.612 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:12.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:11.608723+0000 mon.smithi086 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.154:0/2194779127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:12.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:12.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:12.801611+0000 mgr.smithi086.hkfamo (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:14.159 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 -- 172.21.15.154:0/593073364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 msgr2=0x7f20dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/593073364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc0f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f20cc009fb0 tx=0x7f20cc009b30).stop 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 -- 172.21.15.154:0/593073364 shutdown_connections 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/593073364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc0f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 -- 172.21.15.154:0/593073364 >> 172.21.15.154:0/593073364 conn(0x7f20dc0f1c70 msgr2=0x7f20dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 -- 172.21.15.154:0/593073364 shutdown_connections 2023-11-15T05:02:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.158+0000 7f20e0a5c700 1 -- 172.21.15.154:0/593073364 wait complete. 2023-11-15T05:02:14.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.159+0000 7f20e0a5c700 1 Processor -- start 2023-11-15T05:02:14.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.159+0000 7f20e0a5c700 1 -- start start 2023-11-15T05:02:14.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.159+0000 7f20e0a5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:14.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.159+0000 7f20e0a5c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc106c20 con 0x7f20dc0f6510 2023-11-15T05:02:14.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20da59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:14.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20da59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:48614/0 (socket says 172.21.15.154:48614) 2023-11-15T05:02:14.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20da59c700 1 -- 172.21.15.154:0/1673537392 learned_addr learned my addr 172.21.15.154:0/1673537392 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:14.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20da59c700 1 -- 172.21.15.154:0/1673537392 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc005040 con 0x7f20dc0f6510 2023-11-15T05:02:14.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20da59c700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f20cc013510 tx=0x7f20cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:14.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20cc009320 con 0x7f20dc0f6510 2023-11-15T05:02:14.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20cc013f00 con 0x7f20dc0f6510 2023-11-15T05:02:14.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc106e20 con 0x7f20dc0f6510 2023-11-15T05:02:14.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20cc014d90 con 0x7f20dc0f6510 2023-11-15T05:02:14.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.160+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc1072b0 con 0x7f20dc0f6510 2023-11-15T05:02:14.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.161+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20cc0263a0 con 0x7f20dc0f6510 2023-11-15T05:02:14.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.161+0000 7f20caffd700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 0x7f20c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:14.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.161+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20cc041000 con 0x7f20dc0f6510 2023-11-15T05:02:14.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.161+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc045ad0 con 0x7f20dc0f6510 2023-11-15T05:02:14.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.162+0000 7f20d9d9b700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 0x7f20c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:14.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.162+0000 7f20d9d9b700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 0x7f20c4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f20d00096a0 tx=0x7f20d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:14.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.166+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20cc012920 con 0x7f20dc0f6510 2023-11-15T05:02:14.373 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.372+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20dc0fc050 con 0x7f20dc0f6510 2023-11-15T05:02:14.373 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.373+0000 7f20caffd700 1 -- 172.21.15.154:0/1673537392 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20cc02b020 con 0x7f20dc0f6510 2023-11-15T05:02:14.374 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:14.374 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:14.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.374+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 msgr2=0x7f20c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:14.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.374+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 0x7f20c4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f20d00096a0 tx=0x7f20d0006b40).stop 2023-11-15T05:02:14.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.374+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 msgr2=0x7f20dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:14.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.374+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f20cc013510 tx=0x7f20cc00b040).stop 2023-11-15T05:02:14.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 shutdown_connections 2023-11-15T05:02:14.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f20c402ff60 0x7f20c4032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:14.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 --2- 172.21.15.154:0/1673537392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20dc0f6510 0x7f20dc106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:14.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 >> 172.21.15.154:0/1673537392 conn(0x7f20dc0f1c70 msgr2=0x7f20dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:14.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 shutdown_connections 2023-11-15T05:02:14.377 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:14.375+0000 7f20e0a5c700 1 -- 172.21.15.154:0/1673537392 wait complete. 2023-11-15T05:02:14.379 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:14.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:14.373898+0000 mon.smithi086 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.154:0/1673537392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:15.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:15.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:15.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:14.801917+0000 mgr.smithi086.hkfamo (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:16.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/378328899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 msgr2=0x7fd3a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:16.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/378328899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a80f55c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd390009fb0 tx=0x7fd390009b30).stop 2023-11-15T05:02:16.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/378328899 shutdown_connections 2023-11-15T05:02:16.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/378328899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a80f55c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:16.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/378328899 >> 172.21.15.154:0/378328899 conn(0x7fd3a80f09a0 msgr2=0x7fd3a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:16.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/378328899 shutdown_connections 2023-11-15T05:02:16.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/378328899 wait complete. 2023-11-15T05:02:16.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.839+0000 7fd3ac8d1700 1 Processor -- start 2023-11-15T05:02:16.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3ac8d1700 1 -- start start 2023-11-15T05:02:16.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3ac8d1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3ac8d1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81069d0 con 0x7fd3a80f51a0 2023-11-15T05:02:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3a77fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3a77fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:48624/0 (socket says 172.21.15.154:48624) 2023-11-15T05:02:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3a77fe700 1 -- 172.21.15.154:0/195324916 learned_addr learned my addr 172.21.15.154:0/195324916 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3a77fe700 1 -- 172.21.15.154:0/195324916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390005040 con 0x7fd3a80f51a0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.840+0000 7fd3a77fe700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd390013510 tx=0x7fd39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390009320 con 0x7fd3a80f51a0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8106bd0 con 0x7fd3a80f51a0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd390013f00 con 0x7fd3a80f51a0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390014dc0 con 0x7fd3a80f51a0 2023-11-15T05:02:16.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8107060 con 0x7fd3a80f51a0 2023-11-15T05:02:16.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3900095f0 con 0x7fd3a80f51a0 2023-11-15T05:02:16.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.841+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8045ad0 con 0x7fd3a80f51a0 2023-11-15T05:02:16.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.842+0000 7fd39ffff700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:16.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.842+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd390041040 con 0x7fd3a80f51a0 2023-11-15T05:02:16.848 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.842+0000 7fd3a6ffd700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:16.848 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.842+0000 7fd3a6ffd700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 0x7fd394032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:16.848 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:16.846+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd390014420 con 0x7fd3a80f51a0 2023-11-15T05:02:17.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.053+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3a8107f10 con 0x7fd3a80f51a0 2023-11-15T05:02:17.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.054+0000 7fd39ffff700 1 -- 172.21.15.154:0/195324916 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd390010230 con 0x7fd3a80f51a0 2023-11-15T05:02:17.055 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:17.055 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:17.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 msgr2=0x7fd394032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:17.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 0x7fd394032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).stop 2023-11-15T05:02:17.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 msgr2=0x7fd3a81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:17.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd390013510 tx=0x7fd39000b040).stop 2023-11-15T05:02:17.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 shutdown_connections 2023-11-15T05:02:17.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd39402ff60 0x7fd394032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:17.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 --2- 172.21.15.154:0/195324916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3a80f51a0 0x7fd3a81064e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:17.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 >> 172.21.15.154:0/195324916 conn(0x7fd3a80f09a0 msgr2=0x7fd3a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:17.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 shutdown_connections 2023-11-15T05:02:17.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:17.055+0000 7fd3ac8d1700 1 -- 172.21.15.154:0/195324916 wait complete. 2023-11-15T05:02:17.060 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:16.802239+0000 mgr.smithi086.hkfamo (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:17.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:17.054771+0000 mon.smithi086 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.154:0/195324916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:18.802618+0000 mgr.smithi086.hkfamo (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:19.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:19.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:21.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:20.802898+0000 mgr.smithi086.hkfamo (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:23.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:22.803148+0000 mgr.smithi086.hkfamo (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:24.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 -- 172.21.15.154:0/624709291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 msgr2=0x7f8b200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:24.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 --2- 172.21.15.154:0/624709291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b200f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8b10009fb0 tx=0x7f8b10009b30).stop 2023-11-15T05:02:24.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 -- 172.21.15.154:0/624709291 shutdown_connections 2023-11-15T05:02:24.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 --2- 172.21.15.154:0/624709291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b200f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:24.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 -- 172.21.15.154:0/624709291 >> 172.21.15.154:0/624709291 conn(0x7f8b200f1c70 msgr2=0x7f8b200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:24.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 -- 172.21.15.154:0/624709291 shutdown_connections 2023-11-15T05:02:24.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.268+0000 7f8b282f5700 1 -- 172.21.15.154:0/624709291 wait complete. 2023-11-15T05:02:24.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.269+0000 7f8b282f5700 1 Processor -- start 2023-11-15T05:02:24.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.269+0000 7f8b282f5700 1 -- start start 2023-11-15T05:02:24.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.269+0000 7f8b282f5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:24.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.269+0000 7f8b282f5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20106c20 con 0x7f8b200f6510 2023-11-15T05:02:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b26091700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b26091700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56452/0 (socket says 172.21.15.154:56452) 2023-11-15T05:02:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b26091700 1 -- 172.21.15.154:0/700636221 learned_addr learned my addr 172.21.15.154:0/700636221 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b26091700 1 -- 172.21.15.154:0/700636221 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10005040 con 0x7f8b200f6510 2023-11-15T05:02:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b26091700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8b10014000 tx=0x7f8b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:24.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b100095f0 con 0x7f8b200f6510 2023-11-15T05:02:24.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b20106e20 con 0x7f8b200f6510 2023-11-15T05:02:24.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b10007e70 con 0x7f8b200f6510 2023-11-15T05:02:24.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b10014ea0 con 0x7f8b200f6510 2023-11-15T05:02:24.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.270+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b201072b0 con 0x7f8b200f6510 2023-11-15T05:02:24.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.271+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b10025070 con 0x7f8b200f6510 2023-11-15T05:02:24.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.271+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b20045ad0 con 0x7f8b200f6510 2023-11-15T05:02:24.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.271+0000 7f8b16ffd700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 0x7f8b0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:24.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.271+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b100409f0 con 0x7f8b200f6510 2023-11-15T05:02:24.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.271+0000 7f8b25890700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 0x7f8b0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:24.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.272+0000 7f8b25890700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 0x7f8b0c032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8b1c0096a0 tx=0x7f8b1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:24.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.276+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b1000f4e0 con 0x7f8b200f6510 2023-11-15T05:02:24.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.483+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b200fbf60 con 0x7f8b200f6510 2023-11-15T05:02:24.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.483+0000 7f8b16ffd700 1 -- 172.21.15.154:0/700636221 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b1000f300 con 0x7f8b200f6510 2023-11-15T05:02:24.485 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:24.486 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:24.487 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 msgr2=0x7f8b0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:24.487 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 0x7f8b0c032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8b1c0096a0 tx=0x7f8b1c006b40).stop 2023-11-15T05:02:24.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 msgr2=0x7f8b20106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:24.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8b10014000 tx=0x7f8b1000b040).stop 2023-11-15T05:02:24.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 shutdown_connections 2023-11-15T05:02:24.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8b0c02ffb0 0x7f8b0c032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:24.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 --2- 172.21.15.154:0/700636221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b200f6510 0x7f8b20106730 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:24.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 >> 172.21.15.154:0/700636221 conn(0x7f8b200f1c70 msgr2=0x7f8b200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:24.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 shutdown_connections 2023-11-15T05:02:24.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:24.485+0000 7f8b282f5700 1 -- 172.21.15.154:0/700636221 wait complete. 2023-11-15T05:02:24.489 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:24.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:24.485264+0000 mon.smithi086 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.154:0/700636221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:25.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:24.803460+0000 mgr.smithi086.hkfamo (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:25.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:24.924498+0000 mon.smithi086 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:02:25.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:24.932896+0000 mon.smithi086 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:02:27.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:27.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:26.803810+0000 mgr.smithi086.hkfamo (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:28.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- 172.21.15.154:0/1690073979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 msgr2=0x7f54d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:28.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1690073979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d40f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f54c8009fb0 tx=0x7f54c8009b30).stop 2023-11-15T05:02:28.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- 172.21.15.154:0/1690073979 shutdown_connections 2023-11-15T05:02:28.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1690073979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d40f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:28.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- 172.21.15.154:0/1690073979 >> 172.21.15.154:0/1690073979 conn(0x7f54d40f0b70 msgr2=0x7f54d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:28.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- 172.21.15.154:0/1690073979 shutdown_connections 2023-11-15T05:02:28.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- 172.21.15.154:0/1690073979 wait complete. 2023-11-15T05:02:28.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 Processor -- start 2023-11-15T05:02:28.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.687+0000 7f54db1ee700 1 -- start start 2023-11-15T05:02:28.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54db1ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:28.692 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54db1ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4060280 con 0x7f54d40f5410 2023-11-15T05:02:28.692 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d8f8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:28.692 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d8f8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56476/0 (socket says 172.21.15.154:56476) 2023-11-15T05:02:28.692 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d8f8a700 1 -- 172.21.15.154:0/1937255503 learned_addr learned my addr 172.21.15.154:0/1937255503 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:28.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d8f8a700 1 -- 172.21.15.154:0/1937255503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c8005040 con 0x7f54d40f5410 2023-11-15T05:02:28.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d8f8a700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f54c8013510 tx=0x7f54c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:28.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.688+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54c8009320 con 0x7f54d40f5410 2023-11-15T05:02:28.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d4060480 con 0x7f54d40f5410 2023-11-15T05:02:28.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54c8013f00 con 0x7f54d40f5410 2023-11-15T05:02:28.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54c8014d90 con 0x7f54d40f5410 2023-11-15T05:02:28.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d4109b20 con 0x7f54d40f5410 2023-11-15T05:02:28.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54c80263a0 con 0x7f54d40f5410 2023-11-15T05:02:28.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.689+0000 7f54d17fa700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 0x7f54c4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:28.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.690+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54c80410f0 con 0x7f54d40f5410 2023-11-15T05:02:28.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.690+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d4045ad0 con 0x7f54d40f5410 2023-11-15T05:02:28.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.690+0000 7f54d3fff700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 0x7f54c4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:28.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.690+0000 7f54d3fff700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 0x7f54c4032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f54c00096a0 tx=0x7f54c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:28.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.695+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54c8012920 con 0x7f54d40f5410 2023-11-15T05:02:28.903 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.902+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54d410a1d0 con 0x7f54d40f5410 2023-11-15T05:02:28.904 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.903+0000 7f54d17fa700 1 -- 172.21.15.154:0/1937255503 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54c802b020 con 0x7f54d40f5410 2023-11-15T05:02:28.904 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:28.904 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:28.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 msgr2=0x7f54c4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:28.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 0x7f54c4032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f54c00096a0 tx=0x7f54c0006b40).stop 2023-11-15T05:02:28.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 msgr2=0x7f54d41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:28.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f54c8013510 tx=0x7f54c800b040).stop 2023-11-15T05:02:28.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 shutdown_connections 2023-11-15T05:02:28.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f54c402ffb0 0x7f54c4032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:28.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 --2- 172.21.15.154:0/1937255503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54d40f5410 0x7f54d41093f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:28.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 >> 172.21.15.154:0/1937255503 conn(0x7f54d40f0b70 msgr2=0x7f54d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:28.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 shutdown_connections 2023-11-15T05:02:28.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:28.904+0000 7f54db1ee700 1 -- 172.21.15.154:0/1937255503 wait complete. 2023-11-15T05:02:28.907 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:29.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:28.804144+0000 mgr.smithi086.hkfamo (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:30.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:28.903843+0000 mon.smithi086 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.154:0/1937255503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:30.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:30.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:30.804487+0000 mgr.smithi086.hkfamo (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:31.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 -- 172.21.15.154:0/2906463482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 msgr2=0x7f8f340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:31.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 --2- 172.21.15.154:0/2906463482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f340f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f8f28009fb0 tx=0x7f8f28009b30).stop 2023-11-15T05:02:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 -- 172.21.15.154:0/2906463482 shutdown_connections 2023-11-15T05:02:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 --2- 172.21.15.154:0/2906463482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f340f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 -- 172.21.15.154:0/2906463482 >> 172.21.15.154:0/2906463482 conn(0x7f8f340f0b70 msgr2=0x7f8f340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 -- 172.21.15.154:0/2906463482 shutdown_connections 2023-11-15T05:02:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.403+0000 7f8f39989700 1 -- 172.21.15.154:0/2906463482 wait complete. 2023-11-15T05:02:31.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f39989700 1 Processor -- start 2023-11-15T05:02:31.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f39989700 1 -- start start 2023-11-15T05:02:31.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f39989700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:31.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f39989700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34106c20 con 0x7f8f340f5410 2023-11-15T05:02:31.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f32ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:31.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f32ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56484/0 (socket says 172.21.15.154:56484) 2023-11-15T05:02:31.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f32ffd700 1 -- 172.21.15.154:0/1069215338 learned_addr learned my addr 172.21.15.154:0/1069215338 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:31.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.404+0000 7f8f32ffd700 1 -- 172.21.15.154:0/1069215338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f28005040 con 0x7f8f340f5410 2023-11-15T05:02:31.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f32ffd700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8f28013510 tx=0x7f8f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:31.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f28009320 con 0x7f8f340f5410 2023-11-15T05:02:31.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f34106e20 con 0x7f8f340f5410 2023-11-15T05:02:31.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f28013f00 con 0x7f8f340f5410 2023-11-15T05:02:31.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f28014d90 con 0x7f8f340f5410 2023-11-15T05:02:31.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.405+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f341072b0 con 0x7f8f340f5410 2023-11-15T05:02:31.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f28014420 con 0x7f8f340f5410 2023-11-15T05:02:31.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f34045ad0 con 0x7f8f340f5410 2023-11-15T05:02:31.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f1b7fe700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 0x7f8f1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:31.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f280410c0 con 0x7f8f340f5410 2023-11-15T05:02:31.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f327fc700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 0x7f8f1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:31.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.406+0000 7f8f327fc700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 0x7f8f1c032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8f240096a0 tx=0x7f8f24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:31.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.410+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f280095f0 con 0x7f8f340f5410 2023-11-15T05:02:31.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.618+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f340fae60 con 0x7f8f340f5410 2023-11-15T05:02:31.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.618+0000 7f8f1b7fe700 1 -- 172.21.15.154:0/1069215338 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f2801db70 con 0x7f8f340f5410 2023-11-15T05:02:31.620 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:31.620 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:31.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 msgr2=0x7f8f1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:31.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 0x7f8f1c032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8f240096a0 tx=0x7f8f24006b40).stop 2023-11-15T05:02:31.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 msgr2=0x7f8f34106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:31.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8f28013510 tx=0x7f8f2800b040).stop 2023-11-15T05:02:31.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 shutdown_connections 2023-11-15T05:02:31.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f1c02ffb0 0x7f8f1c032470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:31.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 --2- 172.21.15.154:0/1069215338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f340f5410 0x7f8f34106730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:31.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 >> 172.21.15.154:0/1069215338 conn(0x7f8f340f0b70 msgr2=0x7f8f340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:31.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 shutdown_connections 2023-11-15T05:02:31.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:31.620+0000 7f8f39989700 1 -- 172.21.15.154:0/1069215338 wait complete. 2023-11-15T05:02:31.626 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:32.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:31.619393+0000 mon.smithi086 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.154:0/1069215338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:33.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:33.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:33.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:32.804853+0000 mgr.smithi086.hkfamo (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:34.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 -- 172.21.15.154:0/363380011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 msgr2=0x7ff9680f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:34.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 --2- 172.21.15.154:0/363380011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9680f90d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff95c009fb0 tx=0x7ff95c009b30).stop 2023-11-15T05:02:34.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 -- 172.21.15.154:0/363380011 shutdown_connections 2023-11-15T05:02:34.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 --2- 172.21.15.154:0/363380011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9680f90d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:34.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 -- 172.21.15.154:0/363380011 >> 172.21.15.154:0/363380011 conn(0x7ff9680f0b70 msgr2=0x7ff9680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:34.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 -- 172.21.15.154:0/363380011 shutdown_connections 2023-11-15T05:02:34.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.160+0000 7ff96ebec700 1 -- 172.21.15.154:0/363380011 wait complete. 2023-11-15T05:02:34.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96ebec700 1 Processor -- start 2023-11-15T05:02:34.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96ebec700 1 -- start start 2023-11-15T05:02:34.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96ebec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96ebec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff968106bc0 con 0x7ff968060280 2023-11-15T05:02:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96c988700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96c988700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45698/0 (socket says 172.21.15.154:45698) 2023-11-15T05:02:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.161+0000 7ff96c988700 1 -- 172.21.15.154:0/3725533010 learned_addr learned my addr 172.21.15.154:0/3725533010 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff96c988700 1 -- 172.21.15.154:0/3725533010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c005040 con 0x7ff968060280 2023-11-15T05:02:34.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff96c988700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff95c013510 tx=0x7ff95c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:34.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95c009320 con 0x7ff968060280 2023-11-15T05:02:34.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff968106dc0 con 0x7ff968060280 2023-11-15T05:02:34.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff95c013f00 con 0x7ff968060280 2023-11-15T05:02:34.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff95c014dc0 con 0x7ff968060280 2023-11-15T05:02:34.167 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.162+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff968107250 con 0x7ff968060280 2023-11-15T05:02:34.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff95c014420 con 0x7ff968060280 2023-11-15T05:02:34.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff968045ad0 con 0x7ff968060280 2023-11-15T05:02:34.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff9657fa700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 0x7ff958032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:34.168 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff95c041050 con 0x7ff968060280 2023-11-15T05:02:34.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff967fff700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 0x7ff958032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:34.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.163+0000 7ff967fff700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 0x7ff958032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:34.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.167+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff95c0095f0 con 0x7ff968060280 2023-11-15T05:02:34.372 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.371+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff968108100 con 0x7ff968060280 2023-11-15T05:02:34.373 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.372+0000 7ff9657fa700 1 -- 172.21.15.154:0/3725533010 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff95c01cb10 con 0x7ff968060280 2023-11-15T05:02:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:34.373 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:34.374 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 msgr2=0x7ff958032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:34.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 0x7ff958032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ff9540096a0 tx=0x7ff954006b40).stop 2023-11-15T05:02:34.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 msgr2=0x7ff9681066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:34.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7ff95c013510 tx=0x7ff95c00b040).stop 2023-11-15T05:02:34.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 shutdown_connections 2023-11-15T05:02:34.375 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95802ff60 0x7ff958032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:34.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 --2- 172.21.15.154:0/3725533010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff968060280 0x7ff9681066d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:34.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 >> 172.21.15.154:0/3725533010 conn(0x7ff9680f0b70 msgr2=0x7ff9680f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:34.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 shutdown_connections 2023-11-15T05:02:34.376 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:34.374+0000 7ff96ebec700 1 -- 172.21.15.154:0/3725533010 wait complete. 2023-11-15T05:02:34.377 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:34.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:34.373160+0000 mon.smithi086 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.154:0/3725533010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:35.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:35.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:35.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:34.805125+0000 mgr.smithi086.hkfamo (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:36.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 -- 172.21.15.154:0/2472074697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 msgr2=0x7f8c5c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:36.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/2472074697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c0f90f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8c4c009fb0 tx=0x7f8c4c009b30).stop 2023-11-15T05:02:36.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 -- 172.21.15.154:0/2472074697 shutdown_connections 2023-11-15T05:02:36.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/2472074697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c0f90f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:36.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 -- 172.21.15.154:0/2472074697 >> 172.21.15.154:0/2472074697 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:36.820 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 -- 172.21.15.154:0/2472074697 shutdown_connections 2023-11-15T05:02:36.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.818+0000 7f8c61f6d700 1 -- 172.21.15.154:0/2472074697 wait complete. 2023-11-15T05:02:36.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.819+0000 7f8c61f6d700 1 Processor -- start 2023-11-15T05:02:36.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.819+0000 7f8c61f6d700 1 -- start start 2023-11-15T05:02:36.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.819+0000 7f8c61f6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:36.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c61f6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c106bd0 con 0x7f8c5c060280 2023-11-15T05:02:36.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c5b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:36.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c5b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45718/0 (socket says 172.21.15.154:45718) 2023-11-15T05:02:36.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c5b7fe700 1 -- 172.21.15.154:0/1395989307 learned_addr learned my addr 172.21.15.154:0/1395989307 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:36.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c5b7fe700 1 -- 172.21.15.154:0/1395989307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c005040 con 0x7f8c5c060280 2023-11-15T05:02:36.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c5b7fe700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8c4c013510 tx=0x7f8c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:36.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c4c009320 con 0x7f8c5c060280 2023-11-15T05:02:36.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c106dd0 con 0x7f8c5c060280 2023-11-15T05:02:36.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c4c013f00 con 0x7f8c5c060280 2023-11-15T05:02:36.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c4c014dc0 con 0x7f8c5c060280 2023-11-15T05:02:36.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.820+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c107260 con 0x7f8c5c060280 2023-11-15T05:02:36.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.821+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c4c014420 con 0x7f8c5c060280 2023-11-15T05:02:36.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.821+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c045ad0 con 0x7f8c5c060280 2023-11-15T05:02:36.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.821+0000 7f8c53fff700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:36.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.821+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c4c041070 con 0x7f8c5c060280 2023-11-15T05:02:36.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.822+0000 7f8c5affd700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:36.826 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.822+0000 7f8c5affd700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 0x7f8c48032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8c440096a0 tx=0x7f8c44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:36.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:36.826+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c4c0095f0 con 0x7f8c5c060280 2023-11-15T05:02:37.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.037+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c5c1080b0 con 0x7f8c5c060280 2023-11-15T05:02:37.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.038+0000 7f8c53fff700 1 -- 172.21.15.154:0/1395989307 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c4c01cb10 con 0x7f8c5c060280 2023-11-15T05:02:37.039 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:37.039 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:37.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 msgr2=0x7f8c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:37.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 0x7f8c48032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8c440096a0 tx=0x7f8c44006b40).stop 2023-11-15T05:02:37.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 msgr2=0x7f8c5c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:37.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8c4c013510 tx=0x7f8c4c00b040).stop 2023-11-15T05:02:37.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 shutdown_connections 2023-11-15T05:02:37.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8c4802ff60 0x7f8c48032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:37.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 --2- 172.21.15.154:0/1395989307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c060280 0x7f8c5c1066e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:37.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 >> 172.21.15.154:0/1395989307 conn(0x7f8c5c0f0b70 msgr2=0x7f8c5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:37.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 shutdown_connections 2023-11-15T05:02:37.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:37.040+0000 7f8c61f6d700 1 -- 172.21.15.154:0/1395989307 wait complete. 2023-11-15T05:02:37.043 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:36.805420+0000 mgr.smithi086.hkfamo (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:37.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:37.039216+0000 mon.smithi086 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.154:0/1395989307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:38.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:38.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:39.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:38.805742+0000 mgr.smithi086.hkfamo (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:39.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 -- 172.21.15.154:0/1184946345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 msgr2=0x7fb6c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:39.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/1184946345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c00f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb6b0009fb0 tx=0x7fb6b0009b30).stop 2023-11-15T05:02:39.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 -- 172.21.15.154:0/1184946345 shutdown_connections 2023-11-15T05:02:39.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/1184946345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c00f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:39.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 -- 172.21.15.154:0/1184946345 >> 172.21.15.154:0/1184946345 conn(0x7fb6c00f0b70 msgr2=0x7fb6c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:39.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 -- 172.21.15.154:0/1184946345 shutdown_connections 2023-11-15T05:02:39.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.502+0000 7fb6c79dd700 1 -- 172.21.15.154:0/1184946345 wait complete. 2023-11-15T05:02:39.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.503+0000 7fb6c79dd700 1 Processor -- start 2023-11-15T05:02:39.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.503+0000 7fb6c79dd700 1 -- start start 2023-11-15T05:02:39.506 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.503+0000 7fb6c79dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:39.506 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.503+0000 7fb6c79dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c0105b20 con 0x7fb6c00f5410 2023-11-15T05:02:39.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.503+0000 7fb6c5779700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:39.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c5779700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45738/0 (socket says 172.21.15.154:45738) 2023-11-15T05:02:39.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c5779700 1 -- 172.21.15.154:0/518614690 learned_addr learned my addr 172.21.15.154:0/518614690 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:39.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c5779700 1 -- 172.21.15.154:0/518614690 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b0005040 con 0x7fb6c00f5410 2023-11-15T05:02:39.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c5779700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb6b0013510 tx=0x7fb6b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:39.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b0009320 con 0x7fb6c00f5410 2023-11-15T05:02:39.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6b0013f00 con 0x7fb6c00f5410 2023-11-15T05:02:39.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c0105d20 con 0x7fb6c00f5410 2023-11-15T05:02:39.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b0014d90 con 0x7fb6c00f5410 2023-11-15T05:02:39.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.504+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c01061b0 con 0x7fb6c00f5410 2023-11-15T05:02:39.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.505+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6b0014420 con 0x7fb6c00f5410 2023-11-15T05:02:39.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.505+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c0045ad0 con 0x7fb6c00f5410 2023-11-15T05:02:39.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.505+0000 7fb6b5ffb700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 0x7fb6ac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:39.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.505+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6b00410c0 con 0x7fb6c00f5410 2023-11-15T05:02:39.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.506+0000 7fb6c4f78700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 0x7fb6ac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:39.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.506+0000 7fb6c4f78700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 0x7fb6ac032470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb6bc0096a0 tx=0x7fb6bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:39.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.510+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6b00095f0 con 0x7fb6c00f5410 2023-11-15T05:02:39.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.717+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6c00fae60 con 0x7fb6c00f5410 2023-11-15T05:02:39.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.718+0000 7fb6b5ffb700 1 -- 172.21.15.154:0/518614690 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6b001db70 con 0x7fb6c00f5410 2023-11-15T05:02:39.719 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:39.719 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:39.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 msgr2=0x7fb6ac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:39.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 0x7fb6ac032470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb6bc0096a0 tx=0x7fb6bc006b40).stop 2023-11-15T05:02:39.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 msgr2=0x7fb6c0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:39.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb6b0013510 tx=0x7fb6b000b040).stop 2023-11-15T05:02:39.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 shutdown_connections 2023-11-15T05:02:39.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb6ac02ffb0 0x7fb6ac032470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:39.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 --2- 172.21.15.154:0/518614690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6c00f5410 0x7fb6c0105630 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:39.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 >> 172.21.15.154:0/518614690 conn(0x7fb6c00f0b70 msgr2=0x7fb6c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:39.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 shutdown_connections 2023-11-15T05:02:39.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:39.720+0000 7fb6c79dd700 1 -- 172.21.15.154:0/518614690 wait complete. 2023-11-15T05:02:39.723 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:39.718813+0000 mon.smithi086 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.154:0/518614690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:41.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:41.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:40.806103+0000 mgr.smithi086.hkfamo (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:42.252 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 -- 172.21.15.154:0/1910600601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 msgr2=0x7f2fb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 --2- 172.21.15.154:0/1910600601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb80f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2fac009fb0 tx=0x7f2fac009b30).stop 2023-11-15T05:02:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 -- 172.21.15.154:0/1910600601 shutdown_connections 2023-11-15T05:02:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 --2- 172.21.15.154:0/1910600601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb80f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 -- 172.21.15.154:0/1910600601 >> 172.21.15.154:0/1910600601 conn(0x7f2fb80f1c70 msgr2=0x7f2fb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.251+0000 7f2fbf021700 1 -- 172.21.15.154:0/1910600601 shutdown_connections 2023-11-15T05:02:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.252+0000 7f2fbf021700 1 -- 172.21.15.154:0/1910600601 wait complete. 2023-11-15T05:02:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.252+0000 7f2fbf021700 1 Processor -- start 2023-11-15T05:02:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.252+0000 7f2fbf021700 1 -- start start 2023-11-15T05:02:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.252+0000 7f2fbf021700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.252+0000 7f2fbf021700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb8069e20 con 0x7f2fb80f6510 2023-11-15T05:02:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbcdbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbcdbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45760/0 (socket says 172.21.15.154:45760) 2023-11-15T05:02:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbcdbd700 1 -- 172.21.15.154:0/415363030 learned_addr learned my addr 172.21.15.154:0/415363030 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbcdbd700 1 -- 172.21.15.154:0/415363030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fb80f6510 2023-11-15T05:02:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbcdbd700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2fac013510 tx=0x7f2fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac009320 con 0x7f2fb80f6510 2023-11-15T05:02:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb8069fc0 con 0x7f2fb80f6510 2023-11-15T05:02:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fac013f00 con 0x7f2fb80f6510 2023-11-15T05:02:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac014d90 con 0x7f2fb80f6510 2023-11-15T05:02:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.253+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb806a450 con 0x7f2fb80f6510 2023-11-15T05:02:42.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.254+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fac014420 con 0x7f2fb80f6510 2023-11-15T05:02:42.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.254+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb8059100 con 0x7f2fb80f6510 2023-11-15T05:02:42.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.254+0000 7f2fb57fa700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 0x7f2fa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:42.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.254+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fac040ff0 con 0x7f2fb80f6510 2023-11-15T05:02:42.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.254+0000 7f2fb7fff700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 0x7f2fa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:42.260 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.255+0000 7f2fb7fff700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 0x7f2fa8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f2fa40096a0 tx=0x7f2fa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:42.261 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.259+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fac0095f0 con 0x7f2fb80f6510 2023-11-15T05:02:42.468 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.467+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fb80fbf60 con 0x7f2fb80f6510 2023-11-15T05:02:42.469 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.468+0000 7f2fb57fa700 1 -- 172.21.15.154:0/415363030 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fac01db70 con 0x7f2fb80f6510 2023-11-15T05:02:42.469 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:42.469 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:42.470 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 msgr2=0x7f2fa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:42.471 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 0x7f2fa8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f2fa40096a0 tx=0x7f2fa4006b40).stop 2023-11-15T05:02:42.471 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 msgr2=0x7f2fb806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:42.471 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2fac013510 tx=0x7f2fac00b040).stop 2023-11-15T05:02:42.473 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 shutdown_connections 2023-11-15T05:02:42.473 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fa802ff60 0x7f2fa8032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:42.473 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 --2- 172.21.15.154:0/415363030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb80f6510 0x7f2fb806b9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:42.474 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 >> 172.21.15.154:0/415363030 conn(0x7f2fb80f1c70 msgr2=0x7f2fb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:42.474 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 shutdown_connections 2023-11-15T05:02:42.474 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:42.470+0000 7f2fbf021700 1 -- 172.21.15.154:0/415363030 wait complete. 2023-11-15T05:02:42.475 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:42.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:42.469280+0000 mon.smithi086 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.154:0/415363030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:43.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:43.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:43.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:42.806513+0000 mgr.smithi086.hkfamo (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:44.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 -- 172.21.15.154:0/1977466708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 msgr2=0x7f88b00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:44.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 --2- 172.21.15.154:0/1977466708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b00fb570 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f88a4009fb0 tx=0x7f88a4009b30).stop 2023-11-15T05:02:44.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 -- 172.21.15.154:0/1977466708 shutdown_connections 2023-11-15T05:02:44.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 --2- 172.21.15.154:0/1977466708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b00fb570 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:44.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 -- 172.21.15.154:0/1977466708 >> 172.21.15.154:0/1977466708 conn(0x7f88b00f0b70 msgr2=0x7f88b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:44.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 -- 172.21.15.154:0/1977466708 shutdown_connections 2023-11-15T05:02:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.959+0000 7f88b5717700 1 -- 172.21.15.154:0/1977466708 wait complete. 2023-11-15T05:02:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88b5717700 1 Processor -- start 2023-11-15T05:02:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88b5717700 1 -- start start 2023-11-15T05:02:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88b5717700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88b5717700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b0108dc0 con 0x7f88b0060200 2023-11-15T05:02:44.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:44.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88aeffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54678/0 (socket says 172.21.15.154:54678) 2023-11-15T05:02:44.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88aeffd700 1 -- 172.21.15.154:0/1538957976 learned_addr learned my addr 172.21.15.154:0/1538957976 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:44.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.960+0000 7f88aeffd700 1 -- 172.21.15.154:0/1538957976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4005040 con 0x7f88b0060200 2023-11-15T05:02:44.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88aeffd700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f88a4013510 tx=0x7f88a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:44.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a4009320 con 0x7f88b0060200 2023-11-15T05:02:44.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88a4013f00 con 0x7f88b0060200 2023-11-15T05:02:44.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88a4014dc0 con 0x7f88b0060200 2023-11-15T05:02:44.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b0108fc0 con 0x7f88b0060200 2023-11-15T05:02:44.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.961+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b0109450 con 0x7f88b0060200 2023-11-15T05:02:44.966 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.962+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88a40095f0 con 0x7f88b0060200 2023-11-15T05:02:44.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.962+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b0045ad0 con 0x7f88b0060200 2023-11-15T05:02:44.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.962+0000 7f88977fe700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:44.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.962+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88a4041040 con 0x7f88b0060200 2023-11-15T05:02:44.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.962+0000 7f88ae7fc700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:44.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.963+0000 7f88ae7fc700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 0x7f8898032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f88a00096a0 tx=0x7f88a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:44.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:44.966+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88a4014420 con 0x7f88b0060200 2023-11-15T05:02:45.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.174+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88b010a300 con 0x7f88b0060200 2023-11-15T05:02:45.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.175+0000 7f88977fe700 1 -- 172.21.15.154:0/1538957976 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f88a4010230 con 0x7f88b0060200 2023-11-15T05:02:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:45.176 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:45.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 msgr2=0x7f8898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 0x7f8898032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f88a00096a0 tx=0x7f88a0006b40).stop 2023-11-15T05:02:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 msgr2=0x7f88b01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f88a4013510 tx=0x7f88a400b040).stop 2023-11-15T05:02:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 shutdown_connections 2023-11-15T05:02:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f889802ff60 0x7f8898032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:45.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 --2- 172.21.15.154:0/1538957976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88b0060200 0x7f88b01088d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:45.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 >> 172.21.15.154:0/1538957976 conn(0x7f88b00f0b70 msgr2=0x7f88b00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:45.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 shutdown_connections 2023-11-15T05:02:45.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:45.177+0000 7f88b5717700 1 -- 172.21.15.154:0/1538957976 wait complete. 2023-11-15T05:02:45.180 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:44.806839+0000 mgr.smithi086.hkfamo (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:46.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:45.175881+0000 mon.smithi086 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.154:0/1538957976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:46.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:46.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:47.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 -- 172.21.15.154:0/2676020669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 msgr2=0x7f67000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 --2- 172.21.15.154:0/2676020669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f67000f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f66e8009fb0 tx=0x7f66e8009b30).stop 2023-11-15T05:02:47.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 -- 172.21.15.154:0/2676020669 shutdown_connections 2023-11-15T05:02:47.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 --2- 172.21.15.154:0/2676020669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f67000f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 -- 172.21.15.154:0/2676020669 >> 172.21.15.154:0/2676020669 conn(0x7f67000f1c70 msgr2=0x7f67000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:47.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 -- 172.21.15.154:0/2676020669 shutdown_connections 2023-11-15T05:02:47.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.636+0000 7f670535e700 1 -- 172.21.15.154:0/2676020669 wait complete. 2023-11-15T05:02:47.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.637+0000 7f670535e700 1 Processor -- start 2023-11-15T05:02:47.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.637+0000 7f670535e700 1 -- start start 2023-11-15T05:02:47.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.637+0000 7f670535e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:47.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.637+0000 7f670535e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700106c20 con 0x7f67000f6510 2023-11-15T05:02:47.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66feffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:47.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66feffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54696/0 (socket says 172.21.15.154:54696) 2023-11-15T05:02:47.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66feffd700 1 -- 172.21.15.154:0/620843367 learned_addr learned my addr 172.21.15.154:0/620843367 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:47.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66feffd700 1 -- 172.21.15.154:0/620843367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8005040 con 0x7f67000f6510 2023-11-15T05:02:47.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66feffd700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f66e8013510 tx=0x7f66e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:47.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e8009320 con 0x7f67000f6510 2023-11-15T05:02:47.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6700106e20 con 0x7f67000f6510 2023-11-15T05:02:47.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66e8013f00 con 0x7f67000f6510 2023-11-15T05:02:47.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e8014d90 con 0x7f67000f6510 2023-11-15T05:02:47.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.638+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67001072b0 con 0x7f67000f6510 2023-11-15T05:02:47.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.639+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66e80263a0 con 0x7f67000f6510 2023-11-15T05:02:47.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.639+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700045ad0 con 0x7f67000f6510 2023-11-15T05:02:47.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.639+0000 7f66f77fe700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 0x7f66ec032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:47.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.639+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66e8041370 con 0x7f67000f6510 2023-11-15T05:02:47.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.639+0000 7f66fe7fc700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 0x7f66ec032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:47.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.640+0000 7f66fe7fc700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 0x7f66ec032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f66f00096a0 tx=0x7f66f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:47.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.644+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66e80126f0 con 0x7f67000f6510 2023-11-15T05:02:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:46.807134+0000 mgr.smithi086.hkfamo (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:47.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.851+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67000fbf60 con 0x7f67000f6510 2023-11-15T05:02:47.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.852+0000 7f66f77fe700 1 -- 172.21.15.154:0/620843367 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66e8012330 con 0x7f67000f6510 2023-11-15T05:02:47.853 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:47.853 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:47.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 msgr2=0x7f66ec032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 0x7f66ec032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f66f00096a0 tx=0x7f66f0006b40).stop 2023-11-15T05:02:47.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 msgr2=0x7f6700106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:47.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f66e8013510 tx=0x7f66e800b040).stop 2023-11-15T05:02:47.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 shutdown_connections 2023-11-15T05:02:47.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66ec02ffb0 0x7f66ec032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 --2- 172.21.15.154:0/620843367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67000f6510 0x7f6700106730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:47.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 >> 172.21.15.154:0/620843367 conn(0x7f67000f1c70 msgr2=0x7f67000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:47.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 shutdown_connections 2023-11-15T05:02:47.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:47.853+0000 7f670535e700 1 -- 172.21.15.154:0/620843367 wait complete. 2023-11-15T05:02:47.856 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:48.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:47.852691+0000 mon.smithi086 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.154:0/620843367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:49.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:49.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:49.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:48.807446+0000 mgr.smithi086.hkfamo (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:50.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 -- 172.21.15.154:0/1879412094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 msgr2=0x7fc5a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 --2- 172.21.15.154:0/1879412094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fc598009fb0 tx=0x7fc598009b30).stop 2023-11-15T05:02:50.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 -- 172.21.15.154:0/1879412094 shutdown_connections 2023-11-15T05:02:50.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 --2- 172.21.15.154:0/1879412094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 -- 172.21.15.154:0/1879412094 >> 172.21.15.154:0/1879412094 conn(0x7fc5a80f0b70 msgr2=0x7fc5a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:50.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 -- 172.21.15.154:0/1879412094 shutdown_connections 2023-11-15T05:02:50.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.384+0000 7fc5acb81700 1 -- 172.21.15.154:0/1879412094 wait complete. 2023-11-15T05:02:50.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5acb81700 1 Processor -- start 2023-11-15T05:02:50.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5acb81700 1 -- start start 2023-11-15T05:02:50.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5acb81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:50.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5acb81700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80fb820 con 0x7fc5a80f5410 2023-11-15T05:02:50.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5a659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:50.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5a659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54702/0 (socket says 172.21.15.154:54702) 2023-11-15T05:02:50.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5a659c700 1 -- 172.21.15.154:0/2325288147 learned_addr learned my addr 172.21.15.154:0/2325288147 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:50.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5a659c700 1 -- 172.21.15.154:0/2325288147 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598005040 con 0x7fc5a80f5410 2023-11-15T05:02:50.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.385+0000 7fc5a659c700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc598013510 tx=0x7fc59800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:50.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc598009320 con 0x7fc5a80f5410 2023-11-15T05:02:50.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a80fb9c0 con 0x7fc5a80f5410 2023-11-15T05:02:50.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc598013f00 con 0x7fc5a80f5410 2023-11-15T05:02:50.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc598014d90 con 0x7fc5a80f5410 2023-11-15T05:02:50.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a80fbe50 con 0x7fc5a80f5410 2023-11-15T05:02:50.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a8059100 con 0x7fc5a80f5410 2023-11-15T05:02:50.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.386+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5980095f0 con 0x7fc5a80f5410 2023-11-15T05:02:50.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.387+0000 7fc596ffd700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:50.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.387+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc598041000 con 0x7fc5a80f5410 2023-11-15T05:02:50.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.387+0000 7fc5a5d9b700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:50.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.387+0000 7fc5a5d9b700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 0x7fc590032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc59c0096a0 tx=0x7fc59c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:50.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.391+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc598014420 con 0x7fc5a80f5410 2023-11-15T05:02:50.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.597+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5a80fae60 con 0x7fc5a80f5410 2023-11-15T05:02:50.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.598+0000 7fc596ffd700 1 -- 172.21.15.154:0/2325288147 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc59801d7f0 con 0x7fc5a80f5410 2023-11-15T05:02:50.599 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:50.599 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:50.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 msgr2=0x7fc590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 0x7fc590032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc59c0096a0 tx=0x7fc59c006b40).stop 2023-11-15T05:02:50.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 msgr2=0x7fc5a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:50.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc598013510 tx=0x7fc59800b040).stop 2023-11-15T05:02:50.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 shutdown_connections 2023-11-15T05:02:50.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 --2- 172.21.15.154:0/2325288147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5a80f5410 0x7fc5a80fd3a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:50.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 >> 172.21.15.154:0/2325288147 conn(0x7fc5a80f0b70 msgr2=0x7fc5a805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:50.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 shutdown_connections 2023-11-15T05:02:50.602 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:50.600+0000 7fc5acb81700 1 -- 172.21.15.154:0/2325288147 wait complete. 2023-11-15T05:02:50.603 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:50.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:50.599035+0000 mon.smithi086 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.154:0/2325288147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:51.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:51.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:51.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:50.807752+0000 mgr.smithi086.hkfamo (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:53.100 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- 172.21.15.154:0/2058253970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 msgr2=0x7f13fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:53.100 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 --2- 172.21.15.154:0/2058253970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f13ec009fb0 tx=0x7f13ec009b30).stop 2023-11-15T05:02:53.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- 172.21.15.154:0/2058253970 shutdown_connections 2023-11-15T05:02:53.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 --2- 172.21.15.154:0/2058253970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:53.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- 172.21.15.154:0/2058253970 >> 172.21.15.154:0/2058253970 conn(0x7f13fc0f0b70 msgr2=0x7f13fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:53.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- 172.21.15.154:0/2058253970 shutdown_connections 2023-11-15T05:02:53.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- 172.21.15.154:0/2058253970 wait complete. 2023-11-15T05:02:53.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 Processor -- start 2023-11-15T05:02:53.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.099+0000 7f1400d50700 1 -- start start 2023-11-15T05:02:53.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f1400d50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:53.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f1400d50700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc060240 con 0x7f13fc0f5410 2023-11-15T05:02:53.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:53.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54714/0 (socket says 172.21.15.154:54714) 2023-11-15T05:02:53.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13fa59c700 1 -- 172.21.15.154:0/3543016474 learned_addr learned my addr 172.21.15.154:0/3543016474 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:53.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13fa59c700 1 -- 172.21.15.154:0/3543016474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec005040 con 0x7f13fc0f5410 2023-11-15T05:02:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13fa59c700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f13ec013510 tx=0x7f13ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.100+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ec009320 con 0x7f13fc0f5410 2023-11-15T05:02:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc060440 con 0x7f13fc0f5410 2023-11-15T05:02:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13ec013f00 con 0x7f13fc0f5410 2023-11-15T05:02:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ec014d90 con 0x7f13fc0f5410 2023-11-15T05:02:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc109ae0 con 0x7f13fc0f5410 2023-11-15T05:02:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13ec0095f0 con 0x7f13fc0f5410 2023-11-15T05:02:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13fc045ad0 con 0x7f13fc0f5410 2023-11-15T05:02:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.101+0000 7f13eaffd700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 0x7f13e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.102+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13ec0410a0 con 0x7f13fc0f5410 2023-11-15T05:02:53.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.102+0000 7f13f9d9b700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 0x7f13e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:53.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.102+0000 7f13f9d9b700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 0x7f13e4032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f13f00096a0 tx=0x7f13f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:53.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.106+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13ec06d2b0 con 0x7f13fc0f5410 2023-11-15T05:02:53.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:52.808085+0000 mgr.smithi086.hkfamo (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:53.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.316+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13fc0fae60 con 0x7f13fc0f5410 2023-11-15T05:02:53.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.317+0000 7f13eaffd700 1 -- 172.21.15.154:0/3543016474 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13ec014420 con 0x7f13fc0f5410 2023-11-15T05:02:53.319 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:53.319 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 msgr2=0x7f13e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 0x7f13e4032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f13f00096a0 tx=0x7f13f0006b40).stop 2023-11-15T05:02:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 msgr2=0x7f13fc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f13ec013510 tx=0x7f13ec00b040).stop 2023-11-15T05:02:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 shutdown_connections 2023-11-15T05:02:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13e402ffb0 0x7f13e4032470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 --2- 172.21.15.154:0/3543016474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13fc0f5410 0x7f13fc109430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 >> 172.21.15.154:0/3543016474 conn(0x7f13fc0f0b70 msgr2=0x7f13fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 shutdown_connections 2023-11-15T05:02:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:53.319+0000 7f1400d50700 1 -- 172.21.15.154:0/3543016474 wait complete. 2023-11-15T05:02:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:53.318160+0000 mon.smithi086 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.154:0/3543016474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:54.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:54.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:54.808453+0000 mgr.smithi086.hkfamo (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:55.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 -- 172.21.15.154:0/602227863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 msgr2=0x7efc840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:55.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 --2- 172.21.15.154:0/602227863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc840f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7efc74009fb0 tx=0x7efc74009b30).stop 2023-11-15T05:02:55.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 -- 172.21.15.154:0/602227863 shutdown_connections 2023-11-15T05:02:55.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 --2- 172.21.15.154:0/602227863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc840f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:55.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 -- 172.21.15.154:0/602227863 >> 172.21.15.154:0/602227863 conn(0x7efc840f0b70 msgr2=0x7efc840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:55.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 -- 172.21.15.154:0/602227863 shutdown_connections 2023-11-15T05:02:55.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.852+0000 7efc89e19700 1 -- 172.21.15.154:0/602227863 wait complete. 2023-11-15T05:02:55.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc89e19700 1 Processor -- start 2023-11-15T05:02:55.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc89e19700 1 -- start start 2023-11-15T05:02:55.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc89e19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:55.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc89e19700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84107240 con 0x7efc840f5410 2023-11-15T05:02:55.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc837fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:55.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc837fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60410/0 (socket says 172.21.15.154:60410) 2023-11-15T05:02:55.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc837fe700 1 -- 172.21.15.154:0/4033976783 learned_addr learned my addr 172.21.15.154:0/4033976783 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:55.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.853+0000 7efc837fe700 1 -- 172.21.15.154:0/4033976783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74005040 con 0x7efc840f5410 2023-11-15T05:02:55.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc837fe700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7efc74014000 tx=0x7efc7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:55.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc740095f0 con 0x7efc840f5410 2023-11-15T05:02:55.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc74007e70 con 0x7efc840f5410 2023-11-15T05:02:55.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc84060220 con 0x7efc840f5410 2023-11-15T05:02:55.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc74014ea0 con 0x7efc840f5410 2023-11-15T05:02:55.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.854+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8410bd40 con 0x7efc840f5410 2023-11-15T05:02:55.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc74025070 con 0x7efc840f5410 2023-11-15T05:02:55.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc84045ad0 con 0x7efc840f5410 2023-11-15T05:02:55.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc7bfff700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 0x7efc70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:55.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc74040930 con 0x7efc840f5410 2023-11-15T05:02:55.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc82ffd700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 0x7efc70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:55.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.855+0000 7efc82ffd700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 0x7efc70032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efc6c0096a0 tx=0x7efc6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:55.863 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:55.860+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc7400f4e0 con 0x7efc840f5410 2023-11-15T05:02:56.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.067+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc840fae60 con 0x7efc840f5410 2023-11-15T05:02:56.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.068+0000 7efc7bfff700 1 -- 172.21.15.154:0/4033976783 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc7400f300 con 0x7efc840f5410 2023-11-15T05:02:56.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:56.070 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:56.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 msgr2=0x7efc70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:56.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 0x7efc70032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efc6c0096a0 tx=0x7efc6c006b40).stop 2023-11-15T05:02:56.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 msgr2=0x7efc8410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:56.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7efc74014000 tx=0x7efc7400b040).stop 2023-11-15T05:02:56.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 shutdown_connections 2023-11-15T05:02:56.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7efc7002ff60 0x7efc70032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:56.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 --2- 172.21.15.154:0/4033976783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc840f5410 0x7efc8410b850 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:56.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 >> 172.21.15.154:0/4033976783 conn(0x7efc840f0b70 msgr2=0x7efc840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:56.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 shutdown_connections 2023-11-15T05:02:56.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:56.069+0000 7efc89e19700 1 -- 172.21.15.154:0/4033976783 wait complete. 2023-11-15T05:02:56.075 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:02:56.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:56.068927+0000 mon.smithi086 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.154:0/4033976783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:02:57.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:02:57.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:02:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:56.808759+0000 mgr.smithi086.hkfamo (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:59.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:02:58.808985+0000 mgr.smithi086.hkfamo (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:02:59.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- 172.21.15.154:0/1646168566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 msgr2=0x7ff6640f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:59.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 --2- 172.21.15.154:0/1646168566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff6640f4f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff658009fb0 tx=0x7ff658009b30).stop 2023-11-15T05:02:59.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- 172.21.15.154:0/1646168566 shutdown_connections 2023-11-15T05:02:59.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 --2- 172.21.15.154:0/1646168566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff6640f4f20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:59.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- 172.21.15.154:0/1646168566 >> 172.21.15.154:0/1646168566 conn(0x7ff6640f0260 msgr2=0x7ff6640f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:59.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- 172.21.15.154:0/1646168566 shutdown_connections 2023-11-15T05:02:59.455 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- 172.21.15.154:0/1646168566 wait complete. 2023-11-15T05:02:59.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 Processor -- start 2023-11-15T05:02:59.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.452+0000 7ff66ac13700 1 -- start start 2023-11-15T05:02:59.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff66ac13700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff66ac13700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664107190 con 0x7ff6640f4b00 2023-11-15T05:02:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff6689af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff6689af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60422/0 (socket says 172.21.15.154:60422) 2023-11-15T05:02:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff6689af700 1 -- 172.21.15.154:0/356769815 learned_addr learned my addr 172.21.15.154:0/356769815 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:02:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff6689af700 1 -- 172.21.15.154:0/356769815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658005040 con 0x7ff6640f4b00 2023-11-15T05:02:59.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.453+0000 7ff6689af700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff658013510 tx=0x7ff65800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:59.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff658009320 con 0x7ff6640f4b00 2023-11-15T05:02:59.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff658013f00 con 0x7ff6640f4b00 2023-11-15T05:02:59.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff664107390 con 0x7ff6640f4b00 2023-11-15T05:02:59.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff658014d90 con 0x7ff6640f4b00 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff664107820 con 0x7ff6640f4b00 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.454+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6580263a0 con 0x7ff6640f4b00 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.455+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664045ad0 con 0x7ff6640f4b00 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.455+0000 7ff6617fa700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 0x7ff654032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.455+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff658041360 con 0x7ff6640f4b00 2023-11-15T05:02:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.455+0000 7ff663fff700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 0x7ff654032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:02:59.461 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.455+0000 7ff663fff700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 0x7ff654032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6500096a0 tx=0x7ff650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:02:59.461 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.460+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff658012510 con 0x7ff6640f4b00 2023-11-15T05:02:59.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.665+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6640fa5c0 con 0x7ff6640f4b00 2023-11-15T05:02:59.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.665+0000 7ff6617fa700 1 -- 172.21.15.154:0/356769815 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff658012330 con 0x7ff6640f4b00 2023-11-15T05:02:59.668 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:02:59.668 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:02:59.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 msgr2=0x7ff654032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:59.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 0x7ff654032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff6500096a0 tx=0x7ff650006b40).stop 2023-11-15T05:02:59.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 msgr2=0x7ff664106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:02:59.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff658013510 tx=0x7ff65800b040).stop 2023-11-15T05:02:59.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 shutdown_connections 2023-11-15T05:02:59.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff65402ffb0 0x7ff654032470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:59.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 --2- 172.21.15.154:0/356769815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6640f4b00 0x7ff664106ca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:02:59.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 >> 172.21.15.154:0/356769815 conn(0x7ff6640f0260 msgr2=0x7ff6640f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:02:59.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 shutdown_connections 2023-11-15T05:02:59.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:02:59.667+0000 7ff66ac13700 1 -- 172.21.15.154:0/356769815 wait complete. 2023-11-15T05:02:59.671 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:00.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:02:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:02:59.667638+0000 mon.smithi086 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.154:0/356769815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:01.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:01.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:00.809309+0000 mgr.smithi086.hkfamo (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:02.219 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.216+0000 7f2ee0d83700 1 -- 172.21.15.154:0/4223944752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 msgr2=0x7f2edc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:02.219 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.216+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/4223944752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc0f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2ed0009fb0 tx=0x7f2ed0009b30).stop 2023-11-15T05:03:02.219 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- 172.21.15.154:0/4223944752 shutdown_connections 2023-11-15T05:03:02.219 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/4223944752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc0f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- 172.21.15.154:0/4223944752 >> 172.21.15.154:0/4223944752 conn(0x7f2edc0f1c70 msgr2=0x7f2edc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- 172.21.15.154:0/4223944752 shutdown_connections 2023-11-15T05:03:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- 172.21.15.154:0/4223944752 wait complete. 2023-11-15T05:03:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 Processor -- start 2023-11-15T05:03:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- start start 2023-11-15T05:03:02.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:02.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.217+0000 7f2ee0d83700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc106c20 con 0x7f2edc0f6510 2023-11-15T05:03:02.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2eda59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:02.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2eda59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60446/0 (socket says 172.21.15.154:60446) 2023-11-15T05:03:02.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2eda59c700 1 -- 172.21.15.154:0/2676963857 learned_addr learned my addr 172.21.15.154:0/2676963857 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:02.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2eda59c700 1 -- 172.21.15.154:0/2676963857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0005040 con 0x7f2edc0f6510 2023-11-15T05:03:02.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2eda59c700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2ed0013510 tx=0x7f2ed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed0009320 con 0x7f2edc0f6510 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ed0013f00 con 0x7f2edc0f6510 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc106e20 con 0x7f2edc0f6510 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed0014d90 con 0x7f2edc0f6510 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.218+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc1072b0 con 0x7f2edc0f6510 2023-11-15T05:03:02.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.219+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ed00095f0 con 0x7f2edc0f6510 2023-11-15T05:03:02.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.219+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc045ad0 con 0x7f2edc0f6510 2023-11-15T05:03:02.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.219+0000 7f2ecaffd700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 0x7f2ec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:02.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.219+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ed0041000 con 0x7f2edc0f6510 2023-11-15T05:03:02.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.219+0000 7f2ed9d9b700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 0x7f2ec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:02.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.220+0000 7f2ed9d9b700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 0x7f2ec4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2ecc0096a0 tx=0x7f2ecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:02.226 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.224+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ed001d7f0 con 0x7f2edc0f6510 2023-11-15T05:03:02.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.432+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2edc0fbf60 con 0x7f2edc0f6510 2023-11-15T05:03:02.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.433+0000 7f2ecaffd700 1 -- 172.21.15.154:0/2676963857 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ed0010230 con 0x7f2edc0f6510 2023-11-15T05:03:02.435 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:02.435 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:02.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.434+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 msgr2=0x7f2ec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:02.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 0x7f2ec4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f2ecc0096a0 tx=0x7f2ecc006b40).stop 2023-11-15T05:03:02.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 msgr2=0x7f2edc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:02.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2ed0013510 tx=0x7f2ed000b040).stop 2023-11-15T05:03:02.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 shutdown_connections 2023-11-15T05:03:02.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2ec402ff60 0x7f2ec4032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:02.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 --2- 172.21.15.154:0/2676963857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2edc0f6510 0x7f2edc106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:02.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 >> 172.21.15.154:0/2676963857 conn(0x7f2edc0f1c70 msgr2=0x7f2edc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:02.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 shutdown_connections 2023-11-15T05:03:02.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:02.435+0000 7f2ee0d83700 1 -- 172.21.15.154:0/2676963857 wait complete. 2023-11-15T05:03:02.438 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:02.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:02.435051+0000 mon.smithi086 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.154:0/2676963857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:03.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:03.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:03.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:02.809728+0000 mgr.smithi086.hkfamo (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:04.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.985+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/32025135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 msgr2=0x7fbd480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:04.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.985+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/32025135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd480f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fbd38009fb0 tx=0x7fbd38009b30).stop 2023-11-15T05:03:04.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.985+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/32025135 shutdown_connections 2023-11-15T05:03:04.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.985+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/32025135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd480f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:04.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.985+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/32025135 >> 172.21.15.154:0/32025135 conn(0x7fbd480f09a0 msgr2=0x7fbd480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:04.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/32025135 shutdown_connections 2023-11-15T05:03:04.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/32025135 wait complete. 2023-11-15T05:03:04.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 Processor -- start 2023-11-15T05:03:04.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 -- start start 2023-11-15T05:03:04.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:04.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.986+0000 7fbd4cf0a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48106990 con 0x7fbd480f51a0 2023-11-15T05:03:04.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:04.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58248/0 (socket says 172.21.15.154:58248) 2023-11-15T05:03:04.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd477fe700 1 -- 172.21.15.154:0/4163549444 learned_addr learned my addr 172.21.15.154:0/4163549444 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:04.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd477fe700 1 -- 172.21.15.154:0/4163549444 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38005040 con 0x7fbd480f51a0 2023-11-15T05:03:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd477fe700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fbd38013510 tx=0x7fbd3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd38009320 con 0x7fbd480f51a0 2023-11-15T05:03:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd48106b90 con 0x7fbd480f51a0 2023-11-15T05:03:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd38013f00 con 0x7fbd480f51a0 2023-11-15T05:03:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd38014dc0 con 0x7fbd480f51a0 2023-11-15T05:03:04.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.987+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd48107020 con 0x7fbd480f51a0 2023-11-15T05:03:04.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.988+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd38014420 con 0x7fbd480f51a0 2023-11-15T05:03:04.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.988+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd48045ad0 con 0x7fbd480f51a0 2023-11-15T05:03:04.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.988+0000 7fbd2ffff700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:04.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.988+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd38041110 con 0x7fbd480f51a0 2023-11-15T05:03:04.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.988+0000 7fbd46ffd700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:04.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.989+0000 7fbd46ffd700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbd3c0096a0 tx=0x7fbd3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:04.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:04.993+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd380095f0 con 0x7fbd480f51a0 2023-11-15T05:03:05.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.199+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd48107ed0 con 0x7fbd480f51a0 2023-11-15T05:03:05.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.200+0000 7fbd2ffff700 1 -- 172.21.15.154:0/4163549444 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd3801cb10 con 0x7fbd480f51a0 2023-11-15T05:03:05.201 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:05.201 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:05.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 msgr2=0x7fbd30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:05.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbd3c0096a0 tx=0x7fbd3c006b40).stop 2023-11-15T05:03:05.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 msgr2=0x7fbd481064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:05.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fbd38013510 tx=0x7fbd3800b040).stop 2023-11-15T05:03:05.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 shutdown_connections 2023-11-15T05:03:05.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:05.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 --2- 172.21.15.154:0/4163549444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd480f51a0 0x7fbd481064a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:05.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 >> 172.21.15.154:0/4163549444 conn(0x7fbd480f09a0 msgr2=0x7fbd4805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:05.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 shutdown_connections 2023-11-15T05:03:05.204 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:05.201+0000 7fbd4cf0a700 1 -- 172.21.15.154:0/4163549444 wait complete. 2023-11-15T05:03:05.204 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:04.809887+0000 mgr.smithi086.hkfamo (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:06.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:05.200605+0000 mon.smithi086 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.154:0/4163549444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:06.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:06.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 -- 172.21.15.154:0/520906755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 msgr2=0x7ff9600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 --2- 172.21.15.154:0/520906755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff9600f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ff954009fb0 tx=0x7ff954009b30).stop 2023-11-15T05:03:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 -- 172.21.15.154:0/520906755 shutdown_connections 2023-11-15T05:03:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 --2- 172.21.15.154:0/520906755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff9600f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 -- 172.21.15.154:0/520906755 >> 172.21.15.154:0/520906755 conn(0x7ff9600f0b70 msgr2=0x7ff9600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 -- 172.21.15.154:0/520906755 shutdown_connections 2023-11-15T05:03:07.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.704+0000 7ff966b04700 1 -- 172.21.15.154:0/520906755 wait complete. 2023-11-15T05:03:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.705+0000 7ff966b04700 1 Processor -- start 2023-11-15T05:03:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.705+0000 7ff966b04700 1 -- start start 2023-11-15T05:03:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.705+0000 7ff966b04700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.705+0000 7ff966b04700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff960060240 con 0x7ff9600f5410 2023-11-15T05:03:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff9648a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff9648a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58276/0 (socket says 172.21.15.154:58276) 2023-11-15T05:03:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff9648a0700 1 -- 172.21.15.154:0/1384904342 learned_addr learned my addr 172.21.15.154:0/1384904342 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff9648a0700 1 -- 172.21.15.154:0/1384904342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff954005040 con 0x7ff9600f5410 2023-11-15T05:03:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff9648a0700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff954013510 tx=0x7ff95400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff954009320 con 0x7ff9600f5410 2023-11-15T05:03:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff954013f00 con 0x7ff9600f5410 2023-11-15T05:03:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff960060440 con 0x7ff9600f5410 2023-11-15T05:03:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff954014d90 con 0x7ff9600f5410 2023-11-15T05:03:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.706+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff960109ae0 con 0x7ff9600f5410 2023-11-15T05:03:07.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.707+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9540095f0 con 0x7ff9600f5410 2023-11-15T05:03:07.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.707+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960045ad0 con 0x7ff9600f5410 2023-11-15T05:03:07.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.707+0000 7ff95d7fa700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 0x7ff950032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:07.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.707+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9540410c0 con 0x7ff9600f5410 2023-11-15T05:03:07.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.707+0000 7ff95ffff700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 0x7ff950032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:07.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.708+0000 7ff95ffff700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 0x7ff950032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff94c0096a0 tx=0x7ff94c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:07.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.712+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff954014420 con 0x7ff9600f5410 2023-11-15T05:03:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:06.810177+0000 mgr.smithi086.hkfamo (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:07.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.919+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9600fae60 con 0x7ff9600f5410 2023-11-15T05:03:07.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.920+0000 7ff95d7fa700 1 -- 172.21.15.154:0/1384904342 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff95401d7f0 con 0x7ff9600f5410 2023-11-15T05:03:07.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:07.922 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:07.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 msgr2=0x7ff950032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:07.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 0x7ff950032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff94c0096a0 tx=0x7ff94c006b40).stop 2023-11-15T05:03:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 msgr2=0x7ff960109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff954013510 tx=0x7ff95400b040).stop 2023-11-15T05:03:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 shutdown_connections 2023-11-15T05:03:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff95002ff60 0x7ff950032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 --2- 172.21.15.154:0/1384904342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9600f5410 0x7ff960109430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 >> 172.21.15.154:0/1384904342 conn(0x7ff9600f0b70 msgr2=0x7ff9600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 shutdown_connections 2023-11-15T05:03:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:07.922+0000 7ff966b04700 1 -- 172.21.15.154:0/1384904342 wait complete. 2023-11-15T05:03:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:08.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:07.921159+0000 mon.smithi086 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.154:0/1384904342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:09.304 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:09.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:09.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:08.810558+0000 mgr.smithi086.hkfamo (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:10.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 -- 172.21.15.154:0/4277219918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 msgr2=0x7f23b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:10.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4277219918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b00f6930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f23a0009fb0 tx=0x7f23a0009b30).stop 2023-11-15T05:03:10.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 -- 172.21.15.154:0/4277219918 shutdown_connections 2023-11-15T05:03:10.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4277219918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b00f6930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:10.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 -- 172.21.15.154:0/4277219918 >> 172.21.15.154:0/4277219918 conn(0x7f23b00f1c70 msgr2=0x7f23b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:10.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 -- 172.21.15.154:0/4277219918 shutdown_connections 2023-11-15T05:03:10.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.497+0000 7f23b80b7700 1 -- 172.21.15.154:0/4277219918 wait complete. 2023-11-15T05:03:10.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b80b7700 1 Processor -- start 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b80b7700 1 -- start start 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b80b7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b80b7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0069e20 con 0x7f23b00f6510 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b5e53700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b5e53700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58288/0 (socket says 172.21.15.154:58288) 2023-11-15T05:03:10.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.498+0000 7f23b5e53700 1 -- 172.21.15.154:0/4236099693 learned_addr learned my addr 172.21.15.154:0/4236099693 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:10.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23b5e53700 1 -- 172.21.15.154:0/4236099693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0005040 con 0x7f23b00f6510 2023-11-15T05:03:10.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23b5e53700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f23a0014000 tx=0x7f23a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:10.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a00095f0 con 0x7f23b00f6510 2023-11-15T05:03:10.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b0069fc0 con 0x7f23b00f6510 2023-11-15T05:03:10.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23a0007e70 con 0x7f23b00f6510 2023-11-15T05:03:10.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a0014ea0 con 0x7f23b00f6510 2023-11-15T05:03:10.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.499+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b006a450 con 0x7f23b00f6510 2023-11-15T05:03:10.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.500+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23a0014690 con 0x7f23b00f6510 2023-11-15T05:03:10.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.500+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0059100 con 0x7f23b00f6510 2023-11-15T05:03:10.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.500+0000 7f23a67fc700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 0x7f239c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:10.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.500+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23a0040870 con 0x7f23b00f6510 2023-11-15T05:03:10.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.500+0000 7f23b5652700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 0x7f239c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:10.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.501+0000 7f23b5652700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 0x7f239c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:10.506 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.505+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23a000f300 con 0x7f23b00f6510 2023-11-15T05:03:10.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.710+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23b00fbf60 con 0x7f23b00f6510 2023-11-15T05:03:10.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.711+0000 7f23a67fc700 1 -- 172.21.15.154:0/4236099693 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23a000f300 con 0x7f23b00f6510 2023-11-15T05:03:10.712 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:10.712 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:10.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 msgr2=0x7f239c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:10.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 0x7f239c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).stop 2023-11-15T05:03:10.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 msgr2=0x7f23b006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:10.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f23a0014000 tx=0x7f23a000b040).stop 2023-11-15T05:03:10.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 shutdown_connections 2023-11-15T05:03:10.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f239c02ff60 0x7f239c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:10.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 --2- 172.21.15.154:0/4236099693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23b00f6510 0x7f23b006b9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:10.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 >> 172.21.15.154:0/4236099693 conn(0x7f23b00f1c70 msgr2=0x7f23b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:10.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 shutdown_connections 2023-11-15T05:03:10.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:10.713+0000 7f23b80b7700 1 -- 172.21.15.154:0/4236099693 wait complete. 2023-11-15T05:03:10.717 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:10.712114+0000 mon.smithi086 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.154:0/4236099693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:12.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:12.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:12.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:11 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:10.810975+0000 mgr.smithi086.hkfamo (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:13.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:12.811329+0000 mgr.smithi086.hkfamo (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:13.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 -- 172.21.15.154:0/2089624839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 msgr2=0x7f37fc0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:13.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 --2- 172.21.15.154:0/2089624839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc0f8f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f37f0009fb0 tx=0x7f37f0009b30).stop 2023-11-15T05:03:13.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 -- 172.21.15.154:0/2089624839 shutdown_connections 2023-11-15T05:03:13.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 --2- 172.21.15.154:0/2089624839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc0f8f20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:13.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 -- 172.21.15.154:0/2089624839 >> 172.21.15.154:0/2089624839 conn(0x7f37fc0f09a0 msgr2=0x7f37fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:13.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 -- 172.21.15.154:0/2089624839 shutdown_connections 2023-11-15T05:03:13.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.289+0000 7f3800e2c700 1 -- 172.21.15.154:0/2089624839 wait complete. 2023-11-15T05:03:13.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.290+0000 7f3800e2c700 1 Processor -- start 2023-11-15T05:03:13.293 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.290+0000 7f3800e2c700 1 -- start start 2023-11-15T05:03:13.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.290+0000 7f3800e2c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:13.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.290+0000 7f3800e2c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc1069f0 con 0x7f37fc0601e0 2023-11-15T05:03:13.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37fb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:13.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37fb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:44164/0 (socket says 172.21.15.154:44164) 2023-11-15T05:03:13.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37fb7fe700 1 -- 172.21.15.154:0/3114070723 learned_addr learned my addr 172.21.15.154:0/3114070723 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:13.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37fb7fe700 1 -- 172.21.15.154:0/3114070723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f0005040 con 0x7f37fc0601e0 2023-11-15T05:03:13.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37fb7fe700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f37f0013510 tx=0x7f37f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:13.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f0009320 con 0x7f37fc0601e0 2023-11-15T05:03:13.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37f0013f00 con 0x7f37fc0601e0 2023-11-15T05:03:13.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc106bf0 con 0x7f37fc0601e0 2023-11-15T05:03:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f0014d90 con 0x7f37fc0601e0 2023-11-15T05:03:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.291+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc107080 con 0x7f37fc0601e0 2023-11-15T05:03:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.292+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37f0014420 con 0x7f37fc0601e0 2023-11-15T05:03:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.292+0000 7f37e3fff700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 0x7f37e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.292+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc045ad0 con 0x7f37fc0601e0 2023-11-15T05:03:13.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.292+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37f00410c0 con 0x7f37fc0601e0 2023-11-15T05:03:13.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.292+0000 7f37faffd700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 0x7f37e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:13.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.293+0000 7f37faffd700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 0x7f37e4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f37ec0096a0 tx=0x7f37ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:13.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.297+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37f00095f0 con 0x7f37fc0601e0 2023-11-15T05:03:13.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.505+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37fc107f30 con 0x7f37fc0601e0 2023-11-15T05:03:13.506 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.505+0000 7f37e3fff700 1 -- 172.21.15.154:0/3114070723 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37f001db70 con 0x7f37fc0601e0 2023-11-15T05:03:13.507 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:13.507 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:13.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 msgr2=0x7f37e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:13.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 0x7f37e4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f37ec0096a0 tx=0x7f37ec006b40).stop 2023-11-15T05:03:13.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 msgr2=0x7f37fc106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:13.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f37f0013510 tx=0x7f37f000b040).stop 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 shutdown_connections 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f37e402ff60 0x7f37e4032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 --2- 172.21.15.154:0/3114070723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37fc0601e0 0x7f37fc106500 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 >> 172.21.15.154:0/3114070723 conn(0x7f37fc0f09a0 msgr2=0x7f37fc0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 shutdown_connections 2023-11-15T05:03:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:13.507+0000 7f3800e2c700 1 -- 172.21.15.154:0/3114070723 wait complete. 2023-11-15T05:03:13.510 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:14.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:13.506230+0000 mon.smithi086 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.154:0/3114070723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:14.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:14.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:15.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:14.811705+0000 mgr.smithi086.hkfamo (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:16.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.062+0000 7f6700ddf700 1 -- 172.21.15.154:0/1763166281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 msgr2=0x7f66fc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:16.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.062+0000 7f6700ddf700 1 --2- 172.21.15.154:0/1763166281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f4b60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f66f0009fb0 tx=0x7f66f0009b30).stop 2023-11-15T05:03:16.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.062+0000 7f6700ddf700 1 -- 172.21.15.154:0/1763166281 shutdown_connections 2023-11-15T05:03:16.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.062+0000 7f6700ddf700 1 --2- 172.21.15.154:0/1763166281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f4b60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:16.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.062+0000 7f6700ddf700 1 -- 172.21.15.154:0/1763166281 >> 172.21.15.154:0/1763166281 conn(0x7f66fc0f0260 msgr2=0x7f66fc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:16.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 -- 172.21.15.154:0/1763166281 shutdown_connections 2023-11-15T05:03:16.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 -- 172.21.15.154:0/1763166281 wait complete. 2023-11-15T05:03:16.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 Processor -- start 2023-11-15T05:03:16.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 -- start start 2023-11-15T05:03:16.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:16.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.063+0000 7f6700ddf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0f5770 con 0x7f66fc0f4740 2023-11-15T05:03:16.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:16.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:44174/0 (socket says 172.21.15.154:44174) 2023-11-15T05:03:16.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66fa59c700 1 -- 172.21.15.154:0/702684249 learned_addr learned my addr 172.21.15.154:0/702684249 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:16.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66fa59c700 1 -- 172.21.15.154:0/702684249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0005040 con 0x7f66fc0f4740 2023-11-15T05:03:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66fa59c700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f66f0013510 tx=0x7f66f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0009320 con 0x7f66fc0f4740 2023-11-15T05:03:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66f0013f00 con 0x7f66fc0f4740 2023-11-15T05:03:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc0f5910 con 0x7f66fc0f4740 2023-11-15T05:03:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0014d90 con 0x7f66fc0f4740 2023-11-15T05:03:16.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.064+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc0f5da0 con 0x7f66fc0f4740 2023-11-15T05:03:16.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.065+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66f00095f0 con 0x7f66fc0f4740 2023-11-15T05:03:16.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.065+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc059100 con 0x7f66fc0f4740 2023-11-15T05:03:16.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.065+0000 7f66eaffd700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:16.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.065+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66f00410a0 con 0x7f66fc0f4740 2023-11-15T05:03:16.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.065+0000 7f66f9d9b700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:16.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.066+0000 7f66f9d9b700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 0x7f66e4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f66ec0096a0 tx=0x7f66ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:16.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.070+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66f001d7f0 con 0x7f66fc0f4740 2023-11-15T05:03:16.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.274+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66fc0fbf40 con 0x7f66fc0f4740 2023-11-15T05:03:16.276 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.275+0000 7f66eaffd700 1 -- 172.21.15.154:0/702684249 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66f0010230 con 0x7f66fc0f4740 2023-11-15T05:03:16.276 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:16.276 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:16.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 msgr2=0x7f66e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:16.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 0x7f66e4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f66ec0096a0 tx=0x7f66ec006b40).stop 2023-11-15T05:03:16.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 msgr2=0x7f66fc0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:16.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f66f0013510 tx=0x7f66f000b040).stop 2023-11-15T05:03:16.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 shutdown_connections 2023-11-15T05:03:16.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f66e402ff60 0x7f66e4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:16.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 --2- 172.21.15.154:0/702684249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66fc0f4740 0x7f66fc0f72f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:16.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 >> 172.21.15.154:0/702684249 conn(0x7f66fc0f0260 msgr2=0x7f66fc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:16.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 shutdown_connections 2023-11-15T05:03:16.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:16.277+0000 7f6700ddf700 1 -- 172.21.15.154:0/702684249 wait complete. 2023-11-15T05:03:16.280 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:16.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:16.276265+0000 mon.smithi086 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.154:0/702684249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:17.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:17.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:16.811995+0000 mgr.smithi086.hkfamo (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:18.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 -- 172.21.15.154:0/1216227685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 msgr2=0x7f687c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:18.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 --2- 172.21.15.154:0/1216227685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c0f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f686c009fb0 tx=0x7f686c009b30).stop 2023-11-15T05:03:18.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 -- 172.21.15.154:0/1216227685 shutdown_connections 2023-11-15T05:03:18.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 --2- 172.21.15.154:0/1216227685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c0f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:18.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 -- 172.21.15.154:0/1216227685 >> 172.21.15.154:0/1216227685 conn(0x7f687c0f1c70 msgr2=0x7f687c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:18.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 -- 172.21.15.154:0/1216227685 shutdown_connections 2023-11-15T05:03:18.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.836+0000 7f68833f6700 1 -- 172.21.15.154:0/1216227685 wait complete. 2023-11-15T05:03:18.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f68833f6700 1 Processor -- start 2023-11-15T05:03:18.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f68833f6700 1 -- start start 2023-11-15T05:03:18.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f68833f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:18.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f68833f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c106c20 con 0x7f687c0f6510 2023-11-15T05:03:18.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f6881192700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:18.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f6881192700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:44192/0 (socket says 172.21.15.154:44192) 2023-11-15T05:03:18.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f6881192700 1 -- 172.21.15.154:0/2214783765 learned_addr learned my addr 172.21.15.154:0/2214783765 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:18.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.837+0000 7f6881192700 1 -- 172.21.15.154:0/2214783765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c005040 con 0x7f687c0f6510 2023-11-15T05:03:18.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f6881192700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f686c013510 tx=0x7f686c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:18.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c009320 con 0x7f687c0f6510 2023-11-15T05:03:18.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f686c013f00 con 0x7f687c0f6510 2023-11-15T05:03:18.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c106e20 con 0x7f687c0f6510 2023-11-15T05:03:18.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f686c014d90 con 0x7f687c0f6510 2023-11-15T05:03:18.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.838+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c1072b0 con 0x7f687c0f6510 2023-11-15T05:03:18.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.839+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f686c0263a0 con 0x7f687c0f6510 2023-11-15T05:03:18.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.839+0000 7f6871ffb700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 0x7f6868032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:18.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.839+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c045ad0 con 0x7f687c0f6510 2023-11-15T05:03:18.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.839+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f686c041050 con 0x7f687c0f6510 2023-11-15T05:03:18.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.839+0000 7f6880991700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 0x7f6868032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:18.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.840+0000 7f6880991700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 0x7f6868032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f68780096a0 tx=0x7f6878006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:18.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:18.844+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f686c012920 con 0x7f687c0f6510 2023-11-15T05:03:19.051 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.050+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f687c0fc050 con 0x7f687c0f6510 2023-11-15T05:03:19.052 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.050+0000 7f6871ffb700 1 -- 172.21.15.154:0/2214783765 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f686c012330 con 0x7f687c0f6510 2023-11-15T05:03:19.052 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:19.053 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:19.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 msgr2=0x7f6868032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:19.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 0x7f6868032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f68780096a0 tx=0x7f6878006b40).stop 2023-11-15T05:03:19.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 msgr2=0x7f687c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:19.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f686c013510 tx=0x7f686c00b040).stop 2023-11-15T05:03:19.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 shutdown_connections 2023-11-15T05:03:19.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f686802ffb0 0x7f6868032470 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:19.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 --2- 172.21.15.154:0/2214783765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c0f6510 0x7f687c106730 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:19.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 >> 172.21.15.154:0/2214783765 conn(0x7f687c0f1c70 msgr2=0x7f687c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:19.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 shutdown_connections 2023-11-15T05:03:19.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:19.052+0000 7f68833f6700 1 -- 172.21.15.154:0/2214783765 wait complete. 2023-11-15T05:03:19.058 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:18.812416+0000 mgr.smithi086.hkfamo (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:20.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:19.051521+0000 mon.smithi086 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.154:0/2214783765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:20.422 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:20.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:20.812819+0000 mgr.smithi086.hkfamo (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:21.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 -- 172.21.15.154:0/3045355618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 msgr2=0x7fa0280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3045355618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa0280f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa01c009fb0 tx=0x7fa01c009b30).stop 2023-11-15T05:03:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 -- 172.21.15.154:0/3045355618 shutdown_connections 2023-11-15T05:03:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3045355618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa0280f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 -- 172.21.15.154:0/3045355618 >> 172.21.15.154:0/3045355618 conn(0x7fa0280f1c70 msgr2=0x7fa0280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 -- 172.21.15.154:0/3045355618 shutdown_connections 2023-11-15T05:03:21.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.611+0000 7fa02f52f700 1 -- 172.21.15.154:0/3045355618 wait complete. 2023-11-15T05:03:21.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02f52f700 1 Processor -- start 2023-11-15T05:03:21.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02f52f700 1 -- start start 2023-11-15T05:03:21.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02f52f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:21.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02f52f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa028106c20 con 0x7fa0280f6510 2023-11-15T05:03:21.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02d2cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:21.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02d2cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:44210/0 (socket says 172.21.15.154:44210) 2023-11-15T05:03:21.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02d2cb700 1 -- 172.21.15.154:0/3062599210 learned_addr learned my addr 172.21.15.154:0/3062599210 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:21.617 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02d2cb700 1 -- 172.21.15.154:0/3062599210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c005040 con 0x7fa0280f6510 2023-11-15T05:03:21.617 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa02d2cb700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa01c013510 tx=0x7fa01c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:21.617 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.612+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa01c009320 con 0x7fa0280f6510 2023-11-15T05:03:21.617 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.613+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa01c013f00 con 0x7fa0280f6510 2023-11-15T05:03:21.617 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.613+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa028106e20 con 0x7fa0280f6510 2023-11-15T05:03:21.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.613+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa01c014d90 con 0x7fa0280f6510 2023-11-15T05:03:21.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.613+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0281072b0 con 0x7fa0280f6510 2023-11-15T05:03:21.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa01c014420 con 0x7fa0280f6510 2023-11-15T05:03:21.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa028045ad0 con 0x7fa0280f6510 2023-11-15T05:03:21.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa019ffb700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 0x7fa014032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:21.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa01c040ff0 con 0x7fa0280f6510 2023-11-15T05:03:21.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa02caca700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 0x7fa014032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:21.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.614+0000 7fa02caca700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 0x7fa014032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa0240096a0 tx=0x7fa024006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:21.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.618+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa01c06d2b0 con 0x7fa0280f6510 2023-11-15T05:03:21.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.824+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0280fbf60 con 0x7fa0280f6510 2023-11-15T05:03:21.825 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.824+0000 7fa019ffb700 1 -- 172.21.15.154:0/3062599210 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa01c0095f0 con 0x7fa0280f6510 2023-11-15T05:03:21.825 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:21.826 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:21.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 msgr2=0x7fa014032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 0x7fa014032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa0240096a0 tx=0x7fa024006b40).stop 2023-11-15T05:03:21.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 msgr2=0x7fa028106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:21.827 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fa01c013510 tx=0x7fa01c00b040).stop 2023-11-15T05:03:21.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 shutdown_connections 2023-11-15T05:03:21.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa01402ffb0 0x7fa014032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 --2- 172.21.15.154:0/3062599210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0280f6510 0x7fa028106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:21.828 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 >> 172.21.15.154:0/3062599210 conn(0x7fa0280f1c70 msgr2=0x7fa0280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:21.829 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 shutdown_connections 2023-11-15T05:03:21.831 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:21.826+0000 7fa02f52f700 1 -- 172.21.15.154:0/3062599210 wait complete. 2023-11-15T05:03:21.831 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:22.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:21.825593+0000 mon.smithi086 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.154:0/3062599210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:23.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:23.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:23.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:22.813295+0000 mgr.smithi086.hkfamo (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:24.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/1111411062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 msgr2=0x7f2d380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:24.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/1111411062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d380f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2d20009fb0 tx=0x7f2d20009b30).stop 2023-11-15T05:03:24.332 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/1111411062 shutdown_connections 2023-11-15T05:03:24.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/1111411062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d380f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:24.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/1111411062 >> 172.21.15.154:0/1111411062 conn(0x7f2d380f1c70 msgr2=0x7f2d380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:24.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/1111411062 shutdown_connections 2023-11-15T05:03:24.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.331+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/1111411062 wait complete. 2023-11-15T05:03:24.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d3dcf1700 1 Processor -- start 2023-11-15T05:03:24.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d3dcf1700 1 -- start start 2023-11-15T05:03:24.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d3dcf1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:24.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d3dcf1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d38106c20 con 0x7f2d380f6510 2023-11-15T05:03:24.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d377fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:24.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d377fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56878/0 (socket says 172.21.15.154:56878) 2023-11-15T05:03:24.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.332+0000 7f2d377fe700 1 -- 172.21.15.154:0/2324486574 learned_addr learned my addr 172.21.15.154:0/2324486574 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:24.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d377fe700 1 -- 172.21.15.154:0/2324486574 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d20005040 con 0x7f2d380f6510 2023-11-15T05:03:24.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d377fe700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2d20013510 tx=0x7f2d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:24.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d20009320 con 0x7f2d380f6510 2023-11-15T05:03:24.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d20013f00 con 0x7f2d380f6510 2023-11-15T05:03:24.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d38106e20 con 0x7f2d380f6510 2023-11-15T05:03:24.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d20014d90 con 0x7f2d380f6510 2023-11-15T05:03:24.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.333+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d381072b0 con 0x7f2d380f6510 2023-11-15T05:03:24.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.334+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d200095f0 con 0x7f2d380f6510 2023-11-15T05:03:24.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.334+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d38045ad0 con 0x7f2d380f6510 2023-11-15T05:03:24.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.334+0000 7f2d2ffff700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 0x7f2d24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:24.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.334+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d20041000 con 0x7f2d380f6510 2023-11-15T05:03:24.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.334+0000 7f2d36ffd700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 0x7f2d24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:24.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.335+0000 7f2d36ffd700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 0x7f2d24032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2d280096a0 tx=0x7f2d28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:24.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.338+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d20014420 con 0x7f2d380f6510 2023-11-15T05:03:24.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.546+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d380fbf60 con 0x7f2d380f6510 2023-11-15T05:03:24.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.546+0000 7f2d2ffff700 1 -- 172.21.15.154:0/2324486574 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d20010230 con 0x7f2d380f6510 2023-11-15T05:03:24.547 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:24.548 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:24.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 msgr2=0x7f2d24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:24.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 0x7f2d24032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2d280096a0 tx=0x7f2d28006b40).stop 2023-11-15T05:03:24.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 msgr2=0x7f2d38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:24.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2d20013510 tx=0x7f2d2000b040).stop 2023-11-15T05:03:24.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 shutdown_connections 2023-11-15T05:03:24.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2d2402ff60 0x7f2d24032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:24.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 --2- 172.21.15.154:0/2324486574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d380f6510 0x7f2d38106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:24.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 >> 172.21.15.154:0/2324486574 conn(0x7f2d380f1c70 msgr2=0x7f2d380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:24.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 shutdown_connections 2023-11-15T05:03:24.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:24.548+0000 7f2d3dcf1700 1 -- 172.21.15.154:0/2324486574 wait complete. 2023-11-15T05:03:24.552 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:24.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:24.547603+0000 mon.smithi086 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.154:0/2324486574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:25.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:25.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:25.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:24.813608+0000 mgr.smithi086.hkfamo (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:25.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:24.925991+0000 mon.smithi086 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:03:25.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:24.933718+0000 mon.smithi086 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:03:27.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 -- 172.21.15.154:0/4240388293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 msgr2=0x7fba000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:27.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 --2- 172.21.15.154:0/4240388293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba000f66c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb9f0009fb0 tx=0x7fb9f0009b30).stop 2023-11-15T05:03:27.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 -- 172.21.15.154:0/4240388293 shutdown_connections 2023-11-15T05:03:27.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 --2- 172.21.15.154:0/4240388293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba000f66c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:27.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 -- 172.21.15.154:0/4240388293 >> 172.21.15.154:0/4240388293 conn(0x7fba000f1ac0 msgr2=0x7fba000f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:27.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 -- 172.21.15.154:0/4240388293 shutdown_connections 2023-11-15T05:03:27.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 -- 172.21.15.154:0/4240388293 wait complete. 2023-11-15T05:03:27.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.064+0000 7fba04a2e700 1 Processor -- start 2023-11-15T05:03:27.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fba04a2e700 1 -- start start 2023-11-15T05:03:27.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fba04a2e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:27.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fba04a2e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba00106990 con 0x7fba000f62a0 2023-11-15T05:03:27.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fb9ff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:27.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fb9ff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56892/0 (socket says 172.21.15.154:56892) 2023-11-15T05:03:27.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.065+0000 7fb9ff7fe700 1 -- 172.21.15.154:0/113072358 learned_addr learned my addr 172.21.15.154:0/113072358 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:27.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fb9ff7fe700 1 -- 172.21.15.154:0/113072358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f0005040 con 0x7fba000f62a0 2023-11-15T05:03:27.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fb9ff7fe700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb9f0013510 tx=0x7fb9f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:27.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f0009320 con 0x7fba000f62a0 2023-11-15T05:03:27.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba00106b90 con 0x7fba000f62a0 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9f0013f00 con 0x7fba000f62a0 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9f0014dc0 con 0x7fba000f62a0 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.066+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba00107020 con 0x7fba000f62a0 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.067+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9f001d3d0 con 0x7fba000f62a0 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.067+0000 7fb9e7fff700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.067+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba00045ad0 con 0x7fba000f62a0 2023-11-15T05:03:27.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.067+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9f0041020 con 0x7fba000f62a0 2023-11-15T05:03:27.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.067+0000 7fb9feffd700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:27.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.068+0000 7fb9feffd700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 0x7fb9e8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb9f40096a0 tx=0x7fb9f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:27.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.072+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9f0026490 con 0x7fba000f62a0 2023-11-15T05:03:27.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.278+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba00107ed0 con 0x7fba000f62a0 2023-11-15T05:03:27.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.279+0000 7fb9e7fff700 1 -- 172.21.15.154:0/113072358 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9f00126f0 con 0x7fba000f62a0 2023-11-15T05:03:27.280 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:27.280 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:27.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.280+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 msgr2=0x7fb9e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.280+0000 7fba04a2e700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 0x7fb9e8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb9f40096a0 tx=0x7fb9f4006b40).stop 2023-11-15T05:03:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.280+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 msgr2=0x7fba001064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.280+0000 7fba04a2e700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb9f0013510 tx=0x7fb9f000b040).stop 2023-11-15T05:03:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 shutdown_connections 2023-11-15T05:03:27.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:27.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 --2- 172.21.15.154:0/113072358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba000f62a0 0x7fba001064a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:27.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 >> 172.21.15.154:0/113072358 conn(0x7fba000f1ac0 msgr2=0x7fba000f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:27.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 shutdown_connections 2023-11-15T05:03:27.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:27.281+0000 7fba04a2e700 1 -- 172.21.15.154:0/113072358 wait complete. 2023-11-15T05:03:27.284 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:26.813890+0000 mgr.smithi086.hkfamo (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:27.280013+0000 mon.smithi086 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.154:0/113072358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:28.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:28.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:29.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:28.814196+0000 mgr.smithi086.hkfamo (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:29.904 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 -- 172.21.15.154:0/3899068521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 msgr2=0x7f5c380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:29.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 --2- 172.21.15.154:0/3899068521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c380f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5c28009fb0 tx=0x7f5c28009b30).stop 2023-11-15T05:03:29.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 -- 172.21.15.154:0/3899068521 shutdown_connections 2023-11-15T05:03:29.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 --2- 172.21.15.154:0/3899068521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c380f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:29.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 -- 172.21.15.154:0/3899068521 >> 172.21.15.154:0/3899068521 conn(0x7f5c380f0b70 msgr2=0x7f5c380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:29.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 -- 172.21.15.154:0/3899068521 shutdown_connections 2023-11-15T05:03:29.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.903+0000 7f5c3e599700 1 -- 172.21.15.154:0/3899068521 wait complete. 2023-11-15T05:03:29.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.904+0000 7f5c3e599700 1 Processor -- start 2023-11-15T05:03:29.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.904+0000 7f5c3e599700 1 -- start start 2023-11-15T05:03:29.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.904+0000 7f5c3e599700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:29.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.904+0000 7f5c3e599700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c38106c10 con 0x7f5c380f5410 2023-11-15T05:03:29.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c37fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:29.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c37fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56908/0 (socket says 172.21.15.154:56908) 2023-11-15T05:03:29.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c37fff700 1 -- 172.21.15.154:0/1678214284 learned_addr learned my addr 172.21.15.154:0/1678214284 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:29.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c37fff700 1 -- 172.21.15.154:0/1678214284 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28005040 con 0x7f5c380f5410 2023-11-15T05:03:29.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c37fff700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5c28014000 tx=0x7f5c2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:29.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c280095f0 con 0x7f5c380f5410 2023-11-15T05:03:29.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c38106e10 con 0x7f5c380f5410 2023-11-15T05:03:29.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c28007e70 con 0x7f5c380f5410 2023-11-15T05:03:29.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c28014ea0 con 0x7f5c380f5410 2023-11-15T05:03:29.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.905+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c381072a0 con 0x7f5c380f5410 2023-11-15T05:03:29.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.906+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c38045ad0 con 0x7f5c380f5410 2023-11-15T05:03:29.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.906+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c28025070 con 0x7f5c380f5410 2023-11-15T05:03:29.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.906+0000 7f5c34ff9700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:29.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.907+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c280409c0 con 0x7f5c380f5410 2023-11-15T05:03:29.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.907+0000 7f5c377fe700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:29.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.907+0000 7f5c377fe700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 0x7f5c24032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5c200096a0 tx=0x7f5c20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:29.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:29.911+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c280079c0 con 0x7f5c380f5410 2023-11-15T05:03:30.115 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.115+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c381080f0 con 0x7f5c380f5410 2023-11-15T05:03:30.116 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.116+0000 7f5c34ff9700 1 -- 172.21.15.154:0/1678214284 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c2800f300 con 0x7f5c380f5410 2023-11-15T05:03:30.116 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:30.117 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:30.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 msgr2=0x7f5c24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:30.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 0x7f5c24032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5c200096a0 tx=0x7f5c20006b40).stop 2023-11-15T05:03:30.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 msgr2=0x7f5c38106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:30.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5c28014000 tx=0x7f5c2800b040).stop 2023-11-15T05:03:30.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 shutdown_connections 2023-11-15T05:03:30.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c2402ff60 0x7f5c24032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:30.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 --2- 172.21.15.154:0/1678214284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c380f5410 0x7f5c38106720 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:30.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 >> 172.21.15.154:0/1678214284 conn(0x7f5c380f0b70 msgr2=0x7f5c3805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:30.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 shutdown_connections 2023-11-15T05:03:30.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:30.117+0000 7f5c3e599700 1 -- 172.21.15.154:0/1678214284 wait complete. 2023-11-15T05:03:30.120 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:30.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:30.116458+0000 mon.smithi086 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.154:0/1678214284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:31.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:30.814508+0000 mgr.smithi086.hkfamo (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:31.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:31.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:32.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.659+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3405369869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 msgr2=0x7fa7980f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.659+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3405369869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f9190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa788009fb0 tx=0x7fa788009b30).stop 2023-11-15T05:03:32.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3405369869 shutdown_connections 2023-11-15T05:03:32.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3405369869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f9190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3405369869 >> 172.21.15.154:0/3405369869 conn(0x7fa7980f0b70 msgr2=0x7fa7980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:32.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3405369869 shutdown_connections 2023-11-15T05:03:32.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3405369869 wait complete. 2023-11-15T05:03:32.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 Processor -- start 2023-11-15T05:03:32.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- start start 2023-11-15T05:03:32.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:32.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.660+0000 7fa7a05e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980f5090 con 0x7fa7980601e0 2023-11-15T05:03:32.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa79e380700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:32.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa79e380700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:56914/0 (socket says 172.21.15.154:56914) 2023-11-15T05:03:32.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa79e380700 1 -- 172.21.15.154:0/3024164056 learned_addr learned my addr 172.21.15.154:0/3024164056 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:32.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa79e380700 1 -- 172.21.15.154:0/3024164056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788005040 con 0x7fa7980601e0 2023-11-15T05:03:32.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa79e380700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa788004dd0 tx=0x7fa788009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:32.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa788018070 con 0x7fa7980601e0 2023-11-15T05:03:32.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7980f5230 con 0x7fa7980601e0 2023-11-15T05:03:32.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7980f56c0 con 0x7fa7980601e0 2023-11-15T05:03:32.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa788013410 con 0x7fa7980601e0 2023-11-15T05:03:32.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.661+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa78801c3c0 con 0x7fa7980601e0 2023-11-15T05:03:32.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.662+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa798059100 con 0x7fa7980601e0 2023-11-15T05:03:32.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.662+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa78801d420 con 0x7fa7980601e0 2023-11-15T05:03:32.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.662+0000 7fa78effd700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 0x7fa784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:32.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.662+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7880413a0 con 0x7fa7980601e0 2023-11-15T05:03:32.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.662+0000 7fa79db7f700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 0x7fa784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:32.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.663+0000 7fa79db7f700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 0x7fa784032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa7940096a0 tx=0x7fa794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:32.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.667+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa78801d870 con 0x7fa7980601e0 2023-11-15T05:03:32.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.873+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7980fbf60 con 0x7fa7980601e0 2023-11-15T05:03:32.875 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.874+0000 7fa78effd700 1 -- 172.21.15.154:0/3024164056 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa78801b370 con 0x7fa7980601e0 2023-11-15T05:03:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:32.875 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:32.877 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 msgr2=0x7fa784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.877 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 0x7fa784032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fa7940096a0 tx=0x7fa794006b40).stop 2023-11-15T05:03:32.877 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 msgr2=0x7fa7980f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:32.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa788004dd0 tx=0x7fa788009b30).stop 2023-11-15T05:03:32.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 shutdown_connections 2023-11-15T05:03:32.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa78402ff60 0x7fa784032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 --2- 172.21.15.154:0/3024164056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7980601e0 0x7fa7980f6c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:32.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 >> 172.21.15.154:0/3024164056 conn(0x7fa7980f0b70 msgr2=0x7fa7980f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:32.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 shutdown_connections 2023-11-15T05:03:32.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:32.876+0000 7fa7a05e4700 1 -- 172.21.15.154:0/3024164056 wait complete. 2023-11-15T05:03:32.879 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:32.814888+0000 mgr.smithi086.hkfamo (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:32.875215+0000 mon.smithi086 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.154:0/3024164056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:34.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:34.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:35.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:34.815396+0000 mgr.smithi086.hkfamo (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:35.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 -- 172.21.15.154:0/3798546253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 msgr2=0x7f37600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 --2- 172.21.15.154:0/3798546253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f37600f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3754009fb0 tx=0x7f3754009b30).stop 2023-11-15T05:03:35.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 -- 172.21.15.154:0/3798546253 shutdown_connections 2023-11-15T05:03:35.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 --2- 172.21.15.154:0/3798546253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f37600f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 -- 172.21.15.154:0/3798546253 >> 172.21.15.154:0/3798546253 conn(0x7f37600f1c70 msgr2=0x7f37600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:35.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 -- 172.21.15.154:0/3798546253 shutdown_connections 2023-11-15T05:03:35.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.444+0000 7f37670ad700 1 -- 172.21.15.154:0/3798546253 wait complete. 2023-11-15T05:03:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f37670ad700 1 Processor -- start 2023-11-15T05:03:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f37670ad700 1 -- start start 2023-11-15T05:03:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f37670ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f37670ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760106c20 con 0x7f37600f6510 2023-11-15T05:03:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f3764e49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:35.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f3764e49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36070/0 (socket says 172.21.15.154:36070) 2023-11-15T05:03:35.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f3764e49700 1 -- 172.21.15.154:0/2751175767 learned_addr learned my addr 172.21.15.154:0/2751175767 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:35.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.445+0000 7f3764e49700 1 -- 172.21.15.154:0/2751175767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754005040 con 0x7f37600f6510 2023-11-15T05:03:35.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f3764e49700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3754013510 tx=0x7f375400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:35.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3754009320 con 0x7f37600f6510 2023-11-15T05:03:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3760106e20 con 0x7f37600f6510 2023-11-15T05:03:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3754013f00 con 0x7f37600f6510 2023-11-15T05:03:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3754014d90 con 0x7f37600f6510 2023-11-15T05:03:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.446+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37601072b0 con 0x7f37600f6510 2023-11-15T05:03:35.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37540263a0 con 0x7f37600f6510 2023-11-15T05:03:35.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f375d7fa700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 0x7f3750032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:35.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3754041080 con 0x7f37600f6510 2023-11-15T05:03:35.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3760045ad0 con 0x7f37600f6510 2023-11-15T05:03:35.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f375ffff700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 0x7f3750032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:35.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.447+0000 7f375ffff700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 0x7f3750032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f374c0096a0 tx=0x7f374c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:35.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.452+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3754012920 con 0x7f37600f6510 2023-11-15T05:03:35.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.657+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3760107fd0 con 0x7f37600f6510 2023-11-15T05:03:35.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.658+0000 7f375d7fa700 1 -- 172.21.15.154:0/2751175767 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f375402b020 con 0x7f37600f6510 2023-11-15T05:03:35.659 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:35.659 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:35.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 msgr2=0x7f3750032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 0x7f3750032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f374c0096a0 tx=0x7f374c006b40).stop 2023-11-15T05:03:35.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 msgr2=0x7f3760106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:35.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3754013510 tx=0x7f375400b040).stop 2023-11-15T05:03:35.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 shutdown_connections 2023-11-15T05:03:35.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f375002ffb0 0x7f3750032470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 --2- 172.21.15.154:0/2751175767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37600f6510 0x7f3760106730 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:35.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 >> 172.21.15.154:0/2751175767 conn(0x7f37600f1c70 msgr2=0x7f37600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:35.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 shutdown_connections 2023-11-15T05:03:35.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:35.659+0000 7f37670ad700 1 -- 172.21.15.154:0/2751175767 wait complete. 2023-11-15T05:03:35.663 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:36.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:35.658703+0000 mon.smithi086 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.154:0/2751175767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:36.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:36.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:36.815806+0000 mgr.smithi086.hkfamo (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:38.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 -- 172.21.15.154:0/1204614203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 msgr2=0x7f46e00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:38.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 --2- 172.21.15.154:0/1204614203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e00f8f20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f46d4009fb0 tx=0x7f46d4009b30).stop 2023-11-15T05:03:38.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 -- 172.21.15.154:0/1204614203 shutdown_connections 2023-11-15T05:03:38.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 --2- 172.21.15.154:0/1204614203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e00f8f20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:38.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 -- 172.21.15.154:0/1204614203 >> 172.21.15.154:0/1204614203 conn(0x7f46e00f09a0 msgr2=0x7f46e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:38.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 -- 172.21.15.154:0/1204614203 shutdown_connections 2023-11-15T05:03:38.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.132+0000 7f46e583c700 1 -- 172.21.15.154:0/1204614203 wait complete. 2023-11-15T05:03:38.136 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e583c700 1 Processor -- start 2023-11-15T05:03:38.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e583c700 1 -- start start 2023-11-15T05:03:38.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e583c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:38.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e583c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e0108c10 con 0x7f46e00601e0 2023-11-15T05:03:38.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e483a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:38.137 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e483a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36084/0 (socket says 172.21.15.154:36084) 2023-11-15T05:03:38.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.133+0000 7f46e483a700 1 -- 172.21.15.154:0/3531425906 learned_addr learned my addr 172.21.15.154:0/3531425906 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:38.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46e483a700 1 -- 172.21.15.154:0/3531425906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4005040 con 0x7f46e00601e0 2023-11-15T05:03:38.138 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46e483a700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f46d4013510 tx=0x7f46d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:38.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d4009320 con 0x7f46e00601e0 2023-11-15T05:03:38.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46d4013f00 con 0x7f46e00601e0 2023-11-15T05:03:38.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e0108e10 con 0x7f46e00601e0 2023-11-15T05:03:38.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d4014d90 con 0x7f46e00601e0 2023-11-15T05:03:38.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.134+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e01092a0 con 0x7f46e00601e0 2023-11-15T05:03:38.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.135+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46d40095f0 con 0x7f46e00601e0 2023-11-15T05:03:38.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.135+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e0045ad0 con 0x7f46e00601e0 2023-11-15T05:03:38.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.135+0000 7f46dd7fa700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 0x7f46d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:38.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.135+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46d4041000 con 0x7f46e00601e0 2023-11-15T05:03:38.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.135+0000 7f46dffff700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 0x7f46d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:38.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.136+0000 7f46dffff700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 0x7f46d0032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:38.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.140+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46d4014420 con 0x7f46e00601e0 2023-11-15T05:03:38.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.349+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46e010a150 con 0x7f46e00601e0 2023-11-15T05:03:38.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.350+0000 7f46dd7fa700 1 -- 172.21.15.154:0/3531425906 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46d4010230 con 0x7f46e00601e0 2023-11-15T05:03:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:38.352 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:38.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 msgr2=0x7f46d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:38.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 0x7f46d0032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).stop 2023-11-15T05:03:38.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 msgr2=0x7f46e0108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:38.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f46d4013510 tx=0x7f46d400b040).stop 2023-11-15T05:03:38.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 shutdown_connections 2023-11-15T05:03:38.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f46d002ff60 0x7f46d0032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:38.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 --2- 172.21.15.154:0/3531425906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46e00601e0 0x7f46e0108720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:38.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 >> 172.21.15.154:0/3531425906 conn(0x7f46e00f09a0 msgr2=0x7f46e00f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:38.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 shutdown_connections 2023-11-15T05:03:38.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:38.351+0000 7f46e583c700 1 -- 172.21.15.154:0/3531425906 wait complete. 2023-11-15T05:03:38.355 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:38.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:38.351555+0000 mon.smithi086 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.154:0/3531425906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:39.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:39.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:39.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:38.816182+0000 mgr.smithi086.hkfamo (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:40.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 -- 172.21.15.154:0/1352204886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 msgr2=0x7f78b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:40.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 --2- 172.21.15.154:0/1352204886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b40f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f78a4009fb0 tx=0x7f78a4009b30).stop 2023-11-15T05:03:40.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 -- 172.21.15.154:0/1352204886 shutdown_connections 2023-11-15T05:03:40.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 --2- 172.21.15.154:0/1352204886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b40f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 -- 172.21.15.154:0/1352204886 >> 172.21.15.154:0/1352204886 conn(0x7f78b40f1c70 msgr2=0x7f78b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 -- 172.21.15.154:0/1352204886 shutdown_connections 2023-11-15T05:03:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.965+0000 7f78bbb57700 1 -- 172.21.15.154:0/1352204886 wait complete. 2023-11-15T05:03:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78bbb57700 1 Processor -- start 2023-11-15T05:03:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78bbb57700 1 -- start start 2023-11-15T05:03:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78bbb57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78bbb57700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b4106c20 con 0x7f78b40f6510 2023-11-15T05:03:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78b98f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78b98f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36114/0 (socket says 172.21.15.154:36114) 2023-11-15T05:03:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78b98f3700 1 -- 172.21.15.154:0/2299344883 learned_addr learned my addr 172.21.15.154:0/2299344883 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:40.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.966+0000 7f78b98f3700 1 -- 172.21.15.154:0/2299344883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a4005040 con 0x7f78b40f6510 2023-11-15T05:03:40.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78b98f3700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f78a4013510 tx=0x7f78a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:40.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a4009320 con 0x7f78b40f6510 2023-11-15T05:03:40.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b4106e20 con 0x7f78b40f6510 2023-11-15T05:03:40.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78a4013f00 con 0x7f78b40f6510 2023-11-15T05:03:40.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78a4014d90 con 0x7f78b40f6510 2023-11-15T05:03:40.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.967+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b41072b0 con 0x7f78b40f6510 2023-11-15T05:03:40.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78a40263a0 con 0x7f78b40f6510 2023-11-15T05:03:40.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78aa7fc700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:40.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78a40410c0 con 0x7f78b40f6510 2023-11-15T05:03:40.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b4045ad0 con 0x7f78b40f6510 2023-11-15T05:03:40.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78b90f2700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:40.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.968+0000 7f78b90f2700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 0x7f78a0032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f78b00096a0 tx=0x7f78b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:40.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:40.973+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78a4012920 con 0x7f78b40f6510 2023-11-15T05:03:41.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.177+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78b40fc050 con 0x7f78b40f6510 2023-11-15T05:03:41.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.178+0000 7f78aa7fc700 1 -- 172.21.15.154:0/2299344883 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f78a402b020 con 0x7f78b40f6510 2023-11-15T05:03:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:41.180 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:41.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 msgr2=0x7f78a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:41.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 0x7f78a0032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f78b00096a0 tx=0x7f78b0006b40).stop 2023-11-15T05:03:41.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 msgr2=0x7f78b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:41.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f78a4013510 tx=0x7f78a400b040).stop 2023-11-15T05:03:41.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 shutdown_connections 2023-11-15T05:03:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f78a002ff60 0x7f78a0032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 --2- 172.21.15.154:0/2299344883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b40f6510 0x7f78b4106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 >> 172.21.15.154:0/2299344883 conn(0x7f78b40f1c70 msgr2=0x7f78b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 shutdown_connections 2023-11-15T05:03:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:41.180+0000 7f78bbb57700 1 -- 172.21.15.154:0/2299344883 wait complete. 2023-11-15T05:03:41.184 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:40.816506+0000 mgr.smithi086.hkfamo (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:42.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:41 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:41.180133+0000 mon.smithi086 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.154:0/2299344883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:42.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:42.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:43.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:42.816818+0000 mgr.smithi086.hkfamo (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 -- 172.21.15.154:0/304304024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 msgr2=0x7fd3740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 --2- 172.21.15.154:0/304304024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd364009fb0 tx=0x7fd364009b30).stop 2023-11-15T05:03:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 -- 172.21.15.154:0/304304024 shutdown_connections 2023-11-15T05:03:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 --2- 172.21.15.154:0/304304024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 -- 172.21.15.154:0/304304024 >> 172.21.15.154:0/304304024 conn(0x7fd3740f0b70 msgr2=0x7fd3740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:43.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 -- 172.21.15.154:0/304304024 shutdown_connections 2023-11-15T05:03:43.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.719+0000 7fd37c71e700 1 -- 172.21.15.154:0/304304024 wait complete. 2023-11-15T05:03:43.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37c71e700 1 Processor -- start 2023-11-15T05:03:43.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37c71e700 1 -- start start 2023-11-15T05:03:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37c71e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37c71e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3740fb820 con 0x7fd3740f5410 2023-11-15T05:03:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37a4ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37a4ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37058/0 (socket says 172.21.15.154:37058) 2023-11-15T05:03:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37a4ba700 1 -- 172.21.15.154:0/3363245419 learned_addr learned my addr 172.21.15.154:0/3363245419 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:43.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.720+0000 7fd37a4ba700 1 -- 172.21.15.154:0/3363245419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd364005040 con 0x7fd3740f5410 2023-11-15T05:03:43.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd37a4ba700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd364013510 tx=0x7fd36400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:43.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd364009320 con 0x7fd3740f5410 2023-11-15T05:03:43.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3740fb9c0 con 0x7fd3740f5410 2023-11-15T05:03:43.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd364013f00 con 0x7fd3740f5410 2023-11-15T05:03:43.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd364014d90 con 0x7fd3740f5410 2023-11-15T05:03:43.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.721+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3740fbe50 con 0x7fd3740f5410 2023-11-15T05:03:43.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3640095f0 con 0x7fd3740f5410 2023-11-15T05:03:43.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd374059100 con 0x7fd3740f5410 2023-11-15T05:03:43.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd36affd700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 0x7fd360032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:43.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd364041000 con 0x7fd3740f5410 2023-11-15T05:03:43.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd379cb9700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 0x7fd360032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:43.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.722+0000 7fd379cb9700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 0x7fd360032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd3700096a0 tx=0x7fd370006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:43.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.726+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd364014420 con 0x7fd3740f5410 2023-11-15T05:03:43.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.934+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3740fae60 con 0x7fd3740f5410 2023-11-15T05:03:43.935 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.935+0000 7fd36affd700 1 -- 172.21.15.154:0/3363245419 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd36401d7f0 con 0x7fd3740f5410 2023-11-15T05:03:43.935 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:43.936 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:43.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 msgr2=0x7fd360032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 0x7fd360032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd3700096a0 tx=0x7fd370006b40).stop 2023-11-15T05:03:43.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 msgr2=0x7fd3740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd364013510 tx=0x7fd36400b040).stop 2023-11-15T05:03:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 shutdown_connections 2023-11-15T05:03:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd36002ff60 0x7fd360032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 --2- 172.21.15.154:0/3363245419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3740f5410 0x7fd3740fd3a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 >> 172.21.15.154:0/3363245419 conn(0x7fd3740f0b70 msgr2=0x7fd37405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:43.939 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 shutdown_connections 2023-11-15T05:03:43.939 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:43.936+0000 7fd37c71e700 1 -- 172.21.15.154:0/3363245419 wait complete. 2023-11-15T05:03:43.940 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:44.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:43.935602+0000 mon.smithi086 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.154:0/3363245419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:45.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:45.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:45.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:44.817290+0000 mgr.smithi086.hkfamo (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:46.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- 172.21.15.154:0/1080139372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 msgr2=0x7fad140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/1080139372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad140f55c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fad04009fb0 tx=0x7fad04009b30).stop 2023-11-15T05:03:46.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- 172.21.15.154:0/1080139372 shutdown_connections 2023-11-15T05:03:46.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/1080139372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad140f55c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- 172.21.15.154:0/1080139372 >> 172.21.15.154:0/1080139372 conn(0x7fad140f09a0 msgr2=0x7fad140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:46.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- 172.21.15.154:0/1080139372 shutdown_connections 2023-11-15T05:03:46.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- 172.21.15.154:0/1080139372 wait complete. 2023-11-15T05:03:46.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 Processor -- start 2023-11-15T05:03:46.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- start start 2023-11-15T05:03:46.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:46.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.433+0000 7fad1a5ab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14060240 con 0x7fad140f51a0 2023-11-15T05:03:46.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad195a9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:46.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad195a9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37086/0 (socket says 172.21.15.154:37086) 2023-11-15T05:03:46.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad195a9700 1 -- 172.21.15.154:0/3617200441 learned_addr learned my addr 172.21.15.154:0/3617200441 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:46.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad195a9700 1 -- 172.21.15.154:0/3617200441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04005040 con 0x7fad140f51a0 2023-11-15T05:03:46.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad195a9700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fad04009830 tx=0x7fad0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad04009320 con 0x7fad140f51a0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad14060440 con 0x7fad140f51a0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad040136c0 con 0x7fad140f51a0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.434+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad0401ee90 con 0x7fad140f51a0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.435+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad14109860 con 0x7fad140f51a0 2023-11-15T05:03:46.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.435+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad0401e420 con 0x7fad140f51a0 2023-11-15T05:03:46.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.436+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14045ad0 con 0x7fad140f51a0 2023-11-15T05:03:46.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.436+0000 7fad09ffb700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 0x7fad000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:46.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.436+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad04040f40 con 0x7fad140f51a0 2023-11-15T05:03:46.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.436+0000 7fad18da8700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 0x7fad000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:46.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.436+0000 7fad18da8700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 0x7fad000323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fad100096a0 tx=0x7fad10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:46.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.440+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad0406d2b0 con 0x7fad140f51a0 2023-11-15T05:03:46.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.645+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad1410a070 con 0x7fad140f51a0 2023-11-15T05:03:46.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.646+0000 7fad09ffb700 1 -- 172.21.15.154:0/3617200441 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad040095f0 con 0x7fad140f51a0 2023-11-15T05:03:46.647 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:46.647 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:46.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 msgr2=0x7fad000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 0x7fad000323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fad100096a0 tx=0x7fad10006b40).stop 2023-11-15T05:03:46.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 msgr2=0x7fad141091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:46.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fad04009830 tx=0x7fad0400b040).stop 2023-11-15T05:03:46.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 shutdown_connections 2023-11-15T05:03:46.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fad0002ff10 0x7fad000323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 --2- 172.21.15.154:0/3617200441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140f51a0 0x7fad141091b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:46.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 >> 172.21.15.154:0/3617200441 conn(0x7fad140f09a0 msgr2=0x7fad140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:46.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 shutdown_connections 2023-11-15T05:03:46.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:46.648+0000 7fad1a5ab700 1 -- 172.21.15.154:0/3617200441 wait complete. 2023-11-15T05:03:46.651 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:46.647186+0000 mon.smithi086 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.154:0/3617200441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:46.817717+0000 mgr.smithi086.hkfamo (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:48.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:48.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:49.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 -- 172.21.15.154:0/1821247481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 msgr2=0x7fbc280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 --2- 172.21.15.154:0/1821247481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbc18009fb0 tx=0x7fbc18009b30).stop 2023-11-15T05:03:49.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 -- 172.21.15.154:0/1821247481 shutdown_connections 2023-11-15T05:03:49.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 --2- 172.21.15.154:0/1821247481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 -- 172.21.15.154:0/1821247481 >> 172.21.15.154:0/1821247481 conn(0x7fbc280f0b70 msgr2=0x7fbc280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:49.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 -- 172.21.15.154:0/1821247481 shutdown_connections 2023-11-15T05:03:49.191 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.188+0000 7fbc2d378700 1 -- 172.21.15.154:0/1821247481 wait complete. 2023-11-15T05:03:49.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.189+0000 7fbc2d378700 1 Processor -- start 2023-11-15T05:03:49.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.189+0000 7fbc2d378700 1 -- start start 2023-11-15T05:03:49.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.189+0000 7fbc2d378700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:49.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.189+0000 7fbc2d378700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc280fb820 con 0x7fbc280f5410 2023-11-15T05:03:49.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc26ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37106/0 (socket says 172.21.15.154:37106) 2023-11-15T05:03:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc26ffd700 1 -- 172.21.15.154:0/2935334180 learned_addr learned my addr 172.21.15.154:0/2935334180 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc26ffd700 1 -- 172.21.15.154:0/2935334180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc18005040 con 0x7fbc280f5410 2023-11-15T05:03:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc26ffd700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbc18014000 tx=0x7fbc1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc180095f0 con 0x7fbc280f5410 2023-11-15T05:03:49.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc280fb9c0 con 0x7fbc280f5410 2023-11-15T05:03:49.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc18007e70 con 0x7fbc280f5410 2023-11-15T05:03:49.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc18014ea0 con 0x7fbc280f5410 2023-11-15T05:03:49.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.190+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc280fbe50 con 0x7fbc280f5410 2023-11-15T05:03:49.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.191+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc18025070 con 0x7fbc280f5410 2023-11-15T05:03:49.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.191+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc28059100 con 0x7fbc280f5410 2023-11-15T05:03:49.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.191+0000 7fbc0f7fe700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 0x7fbc10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:49.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.191+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc180409e0 con 0x7fbc280f5410 2023-11-15T05:03:49.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.192+0000 7fbc267fc700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 0x7fbc10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:49.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.192+0000 7fbc267fc700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 0x7fbc10032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbc1c0096a0 tx=0x7fbc1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:49.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.196+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc1800f4e0 con 0x7fbc280f5410 2023-11-15T05:03:49.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:48.818122+0000 mgr.smithi086.hkfamo (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:49.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.404+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc28045ad0 con 0x7fbc280f5410 2023-11-15T05:03:49.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.404+0000 7fbc0f7fe700 1 -- 172.21.15.154:0/2935334180 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc1800f300 con 0x7fbc280f5410 2023-11-15T05:03:49.406 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:49.406 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:49.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 msgr2=0x7fbc10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 0x7fbc10032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbc1c0096a0 tx=0x7fbc1c006b40).stop 2023-11-15T05:03:49.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 msgr2=0x7fbc280fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:49.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbc18014000 tx=0x7fbc1800b040).stop 2023-11-15T05:03:49.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 shutdown_connections 2023-11-15T05:03:49.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbc1002ff60 0x7fbc10032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 --2- 172.21.15.154:0/2935334180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc280f5410 0x7fbc280fd3a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:49.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 >> 172.21.15.154:0/2935334180 conn(0x7fbc280f0b70 msgr2=0x7fbc2805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:49.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 shutdown_connections 2023-11-15T05:03:49.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:49.406+0000 7fbc2d378700 1 -- 172.21.15.154:0/2935334180 wait complete. 2023-11-15T05:03:49.410 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:50.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:49.405482+0000 mon.smithi086 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.154:0/2935334180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:50.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:50.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:51.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:50.818519+0000 mgr.smithi086.hkfamo (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:51.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.905+0000 7f7b7b869700 1 -- 172.21.15.154:0/1257408311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 msgr2=0x7f7b740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.905+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1257408311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b740f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f7b64009fb0 tx=0x7f7b64009b30).stop 2023-11-15T05:03:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 -- 172.21.15.154:0/1257408311 shutdown_connections 2023-11-15T05:03:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1257408311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b740f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 -- 172.21.15.154:0/1257408311 >> 172.21.15.154:0/1257408311 conn(0x7f7b740f0b70 msgr2=0x7f7b740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 -- 172.21.15.154:0/1257408311 shutdown_connections 2023-11-15T05:03:51.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 -- 172.21.15.154:0/1257408311 wait complete. 2023-11-15T05:03:51.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 Processor -- start 2023-11-15T05:03:51.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.906+0000 7f7b7b869700 1 -- start start 2023-11-15T05:03:51.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b7b869700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:51.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b7b869700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74060220 con 0x7f7b740f5410 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b79605700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b79605700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37134/0 (socket says 172.21.15.154:37134) 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b79605700 1 -- 172.21.15.154:0/1352458817 learned_addr learned my addr 172.21.15.154:0/1352458817 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b79605700 1 -- 172.21.15.154:0/1352458817 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64005040 con 0x7f7b740f5410 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b79605700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7b64014000 tx=0x7f7b6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:51.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b640095f0 con 0x7f7b740f5410 2023-11-15T05:03:51.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b64007e70 con 0x7f7b740f5410 2023-11-15T05:03:51.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b74060420 con 0x7f7b740f5410 2023-11-15T05:03:51.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.907+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b64014ea0 con 0x7f7b740f5410 2023-11-15T05:03:51.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.908+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b74109b00 con 0x7f7b740f5410 2023-11-15T05:03:51.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.908+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b64025070 con 0x7f7b740f5410 2023-11-15T05:03:51.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.908+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b74059100 con 0x7f7b740f5410 2023-11-15T05:03:51.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.909+0000 7f7b69ffb700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 0x7f7b60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:51.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.909+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b640409e0 con 0x7f7b740f5410 2023-11-15T05:03:51.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.909+0000 7f7b78e04700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 0x7f7b60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:51.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.909+0000 7f7b78e04700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 0x7f7b60032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7b700096a0 tx=0x7f7b70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:51.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:51.913+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b6400f4e0 con 0x7f7b740f5410 2023-11-15T05:03:52.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.123+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b74045ad0 con 0x7f7b740f5410 2023-11-15T05:03:52.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.124+0000 7f7b69ffb700 1 -- 172.21.15.154:0/1352458817 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b6400f300 con 0x7f7b740f5410 2023-11-15T05:03:52.125 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:52.125 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:52.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.125+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 msgr2=0x7f7b60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:52.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.125+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 0x7f7b60032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7b700096a0 tx=0x7f7b70006b40).stop 2023-11-15T05:03:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 msgr2=0x7f7b74109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7b64014000 tx=0x7f7b6400b040).stop 2023-11-15T05:03:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 shutdown_connections 2023-11-15T05:03:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b6002ff60 0x7f7b60032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 --2- 172.21.15.154:0/1352458817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b740f5410 0x7f7b74109450 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:52.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 >> 172.21.15.154:0/1352458817 conn(0x7f7b740f0b70 msgr2=0x7f7b740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:52.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 shutdown_connections 2023-11-15T05:03:52.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:52.126+0000 7f7b7b869700 1 -- 172.21.15.154:0/1352458817 wait complete. 2023-11-15T05:03:52.128 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:52.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:52.124745+0000 mon.smithi086 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.154:0/1352458817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:53.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:53.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:53.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:52.818819+0000 mgr.smithi086.hkfamo (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:54.623 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568037345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 msgr2=0x7fc1a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568037345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a00f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc194009fb0 tx=0x7fc194009b30).stop 2023-11-15T05:03:54.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568037345 shutdown_connections 2023-11-15T05:03:54.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568037345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a00f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568037345 >> 172.21.15.154:0/568037345 conn(0x7fc1a00f0b70 msgr2=0x7fc1a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568037345 shutdown_connections 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.622+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568037345 wait complete. 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a6e14700 1 Processor -- start 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a6e14700 1 -- start start 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a6e14700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:54.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a6e14700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a0060240 con 0x7fc1a00f5410 2023-11-15T05:03:54.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a4bb0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:54.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a4bb0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39088/0 (socket says 172.21.15.154:39088) 2023-11-15T05:03:54.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a4bb0700 1 -- 172.21.15.154:0/568368181 learned_addr learned my addr 172.21.15.154:0/568368181 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:54.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.623+0000 7fc1a4bb0700 1 -- 172.21.15.154:0/568368181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194005040 con 0x7fc1a00f5410 2023-11-15T05:03:54.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc1a4bb0700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc194013510 tx=0x7fc19400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:54.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc194009320 con 0x7fc1a00f5410 2023-11-15T05:03:54.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a0060440 con 0x7fc1a00f5410 2023-11-15T05:03:54.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc194013f00 con 0x7fc1a00f5410 2023-11-15T05:03:54.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc194014d90 con 0x7fc1a00f5410 2023-11-15T05:03:54.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a0109ae0 con 0x7fc1a00f5410 2023-11-15T05:03:54.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.624+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1940263a0 con 0x7fc1a00f5410 2023-11-15T05:03:54.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.625+0000 7fc19d7fa700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 0x7fc190032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:54.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.625+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a0045ad0 con 0x7fc1a00f5410 2023-11-15T05:03:54.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.625+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc194041060 con 0x7fc1a00f5410 2023-11-15T05:03:54.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.625+0000 7fc19ffff700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 0x7fc190032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:54.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.625+0000 7fc19ffff700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 0x7fc190032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc18c0096a0 tx=0x7fc18c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:54.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.629+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc194012920 con 0x7fc1a00f5410 2023-11-15T05:03:54.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.842+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1a00faf50 con 0x7fc1a00f5410 2023-11-15T05:03:54.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.842+0000 7fc19d7fa700 1 -- 172.21.15.154:0/568368181 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc19402b020 con 0x7fc1a00f5410 2023-11-15T05:03:54.843 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:54.844 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:54.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 msgr2=0x7fc190032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 0x7fc190032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc18c0096a0 tx=0x7fc18c006b40).stop 2023-11-15T05:03:54.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 msgr2=0x7fc1a0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:54.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc194013510 tx=0x7fc19400b040).stop 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 shutdown_connections 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc19002ffb0 0x7fc190032470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 --2- 172.21.15.154:0/568368181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1a00f5410 0x7fc1a0109430 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 >> 172.21.15.154:0/568368181 conn(0x7fc1a00f0b70 msgr2=0x7fc1a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 shutdown_connections 2023-11-15T05:03:54.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:54.844+0000 7fc1a6e14700 1 -- 172.21.15.154:0/568368181 wait complete. 2023-11-15T05:03:54.847 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:54.819168+0000 mgr.smithi086.hkfamo (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:54.843555+0000 mon.smithi086 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.154:0/568368181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:56.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:56.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:57.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 -- 172.21.15.154:0/1531479036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 msgr2=0x7fd9380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/1531479036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd92c009fb0 tx=0x7fd92c009b30).stop 2023-11-15T05:03:57.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 -- 172.21.15.154:0/1531479036 shutdown_connections 2023-11-15T05:03:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/1531479036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 -- 172.21.15.154:0/1531479036 >> 172.21.15.154:0/1531479036 conn(0x7fd9380f0b70 msgr2=0x7fd9380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 -- 172.21.15.154:0/1531479036 shutdown_connections 2023-11-15T05:03:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.397+0000 7fd93ee4a700 1 -- 172.21.15.154:0/1531479036 wait complete. 2023-11-15T05:03:57.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93ee4a700 1 Processor -- start 2023-11-15T05:03:57.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93ee4a700 1 -- start start 2023-11-15T05:03:57.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93ee4a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:57.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93ee4a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380f9a00 con 0x7fd9380f5410 2023-11-15T05:03:57.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93cbe6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:57.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93cbe6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39100/0 (socket says 172.21.15.154:39100) 2023-11-15T05:03:57.402 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93cbe6700 1 -- 172.21.15.154:0/3503127791 learned_addr learned my addr 172.21.15.154:0/3503127791 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:03:57.402 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.398+0000 7fd93cbe6700 1 -- 172.21.15.154:0/3503127791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c005040 con 0x7fd9380f5410 2023-11-15T05:03:57.402 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd93cbe6700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd92c013510 tx=0x7fd92c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:57.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd92c009320 con 0x7fd9380f5410 2023-11-15T05:03:57.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9380f9ba0 con 0x7fd9380f5410 2023-11-15T05:03:57.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd92c013f00 con 0x7fd9380f5410 2023-11-15T05:03:57.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd92c014d90 con 0x7fd9380f5410 2023-11-15T05:03:57.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.399+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9380fa030 con 0x7fd9380f5410 2023-11-15T05:03:57.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd92c014420 con 0x7fd9380f5410 2023-11-15T05:03:57.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938059100 con 0x7fd9380f5410 2023-11-15T05:03:57.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd9357fa700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 0x7fd928032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:03:57.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd92c041070 con 0x7fd9380f5410 2023-11-15T05:03:57.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd937fff700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 0x7fd928032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:03:57.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.400+0000 7fd937fff700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 0x7fd928032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd9240096a0 tx=0x7fd924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:03:57.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.404+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd92c0095f0 con 0x7fd9380f5410 2023-11-15T05:03:57.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.610+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9380fbf90 con 0x7fd9380f5410 2023-11-15T05:03:57.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.611+0000 7fd9357fa700 1 -- 172.21.15.154:0/3503127791 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd92c01db70 con 0x7fd9380f5410 2023-11-15T05:03:57.612 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:03:57.612 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:03:57.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 msgr2=0x7fd928032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 0x7fd928032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd9240096a0 tx=0x7fd924006b40).stop 2023-11-15T05:03:57.613 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 msgr2=0x7fd9380fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:03:57.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd92c013510 tx=0x7fd92c00b040).stop 2023-11-15T05:03:57.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 shutdown_connections 2023-11-15T05:03:57.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd92802ff60 0x7fd928032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 --2- 172.21.15.154:0/3503127791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9380f5410 0x7fd9380fb580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:03:57.614 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.612+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 >> 172.21.15.154:0/3503127791 conn(0x7fd9380f0b70 msgr2=0x7fd9380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:03:57.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.613+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 shutdown_connections 2023-11-15T05:03:57.615 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:03:57.613+0000 7fd93ee4a700 1 -- 172.21.15.154:0/3503127791 wait complete. 2023-11-15T05:03:57.615 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:03:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:56.819569+0000 mgr.smithi086.hkfamo (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:03:58.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:03:57.611894+0000 mon.smithi086 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.154:0/3503127791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:03:58.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:03:58.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:03:59.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:03:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:03:58.819906+0000 mgr.smithi086.hkfamo (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:00.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- 172.21.15.154:0/4201740247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 msgr2=0x7f24ac0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 --2- 172.21.15.154:0/4201740247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac0f8f20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f249c009fb0 tx=0x7f249c009b30).stop 2023-11-15T05:04:00.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- 172.21.15.154:0/4201740247 shutdown_connections 2023-11-15T05:04:00.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 --2- 172.21.15.154:0/4201740247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac0f8f20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- 172.21.15.154:0/4201740247 >> 172.21.15.154:0/4201740247 conn(0x7f24ac0f09a0 msgr2=0x7f24ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:00.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- 172.21.15.154:0/4201740247 shutdown_connections 2023-11-15T05:04:00.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- 172.21.15.154:0/4201740247 wait complete. 2023-11-15T05:04:00.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 Processor -- start 2023-11-15T05:04:00.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- start start 2023-11-15T05:04:00.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:00.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.138+0000 7f24ab59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac1069f0 con 0x7f24ac0601e0 2023-11-15T05:04:00.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24aa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:00.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24aa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39104/0 (socket says 172.21.15.154:39104) 2023-11-15T05:04:00.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24aa59c700 1 -- 172.21.15.154:0/2501596571 learned_addr learned my addr 172.21.15.154:0/2501596571 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:00.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24aa59c700 1 -- 172.21.15.154:0/2501596571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c005040 con 0x7f24ac0601e0 2023-11-15T05:04:00.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24aa59c700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f249c013510 tx=0x7f249c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:00.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f249c009320 con 0x7f24ac0601e0 2023-11-15T05:04:00.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f249c013f00 con 0x7f24ac0601e0 2023-11-15T05:04:00.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f249c014d90 con 0x7f24ac0601e0 2023-11-15T05:04:00.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac106bf0 con 0x7f24ac0601e0 2023-11-15T05:04:00.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.139+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac107080 con 0x7f24ac0601e0 2023-11-15T05:04:00.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.140+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f249c0095f0 con 0x7f24ac0601e0 2023-11-15T05:04:00.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.140+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ac045ad0 con 0x7f24ac0601e0 2023-11-15T05:04:00.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.140+0000 7f249affd700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 0x7f2494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:00.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.140+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f249c041000 con 0x7f24ac0601e0 2023-11-15T05:04:00.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.141+0000 7f24a9d9b700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 0x7f2494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:00.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.141+0000 7f24a9d9b700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 0x7f2494032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f24a00096a0 tx=0x7f24a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:00.148 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.145+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f249c01d7f0 con 0x7f24ac0601e0 2023-11-15T05:04:00.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.351+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24ac107f30 con 0x7f24ac0601e0 2023-11-15T05:04:00.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.352+0000 7f249affd700 1 -- 172.21.15.154:0/2501596571 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f249c010230 con 0x7f24ac0601e0 2023-11-15T05:04:00.353 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:00.353 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:00.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 msgr2=0x7f2494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 0x7f2494032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f24a00096a0 tx=0x7f24a0006b40).stop 2023-11-15T05:04:00.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 msgr2=0x7f24ac106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:00.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f249c013510 tx=0x7f249c00b040).stop 2023-11-15T05:04:00.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 shutdown_connections 2023-11-15T05:04:00.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f249402ff60 0x7f2494032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 --2- 172.21.15.154:0/2501596571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24ac0601e0 0x7f24ac106500 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:00.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 >> 172.21.15.154:0/2501596571 conn(0x7f24ac0f09a0 msgr2=0x7f24ac0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:00.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 shutdown_connections 2023-11-15T05:04:00.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:00.353+0000 7f24ab59e700 1 -- 172.21.15.154:0/2501596571 wait complete. 2023-11-15T05:04:00.356 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:00.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:00.352752+0000 mon.smithi086 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.154:0/2501596571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:01.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:01 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:00.820254+0000 mgr.smithi086.hkfamo (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:01.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:01.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:02.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 -- 172.21.15.154:0/382611720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 msgr2=0x7f4eb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:02.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 --2- 172.21.15.154:0/382611720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb40f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f4ea8009fb0 tx=0x7f4ea8009b30).stop 2023-11-15T05:04:02.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 -- 172.21.15.154:0/382611720 shutdown_connections 2023-11-15T05:04:02.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 --2- 172.21.15.154:0/382611720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb40f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:02.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 -- 172.21.15.154:0/382611720 >> 172.21.15.154:0/382611720 conn(0x7f4eb40f0b70 msgr2=0x7f4eb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 -- 172.21.15.154:0/382611720 shutdown_connections 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.924+0000 7f4ebb205700 1 -- 172.21.15.154:0/382611720 wait complete. 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.925+0000 7f4ebb205700 1 Processor -- start 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.925+0000 7f4ebb205700 1 -- start start 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.925+0000 7f4ebb205700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.925+0000 7f4ebb205700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb4060280 con 0x7f4eb40f5410 2023-11-15T05:04:02.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb8fa1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:02.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb8fa1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39132/0 (socket says 172.21.15.154:39132) 2023-11-15T05:04:02.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb8fa1700 1 -- 172.21.15.154:0/880774564 learned_addr learned my addr 172.21.15.154:0/880774564 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:02.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb8fa1700 1 -- 172.21.15.154:0/880774564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8005040 con 0x7f4eb40f5410 2023-11-15T05:04:02.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb8fa1700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4ea8013510 tx=0x7f4ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:02.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea8009320 con 0x7f4eb40f5410 2023-11-15T05:04:02.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb4060480 con 0x7f4eb40f5410 2023-11-15T05:04:02.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ea8013f00 con 0x7f4eb40f5410 2023-11-15T05:04:02.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea8014d90 con 0x7f4eb40f5410 2023-11-15T05:04:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.926+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb4109b20 con 0x7f4eb40f5410 2023-11-15T05:04:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.927+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ea8014420 con 0x7f4eb40f5410 2023-11-15T05:04:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.927+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb4045ad0 con 0x7f4eb40f5410 2023-11-15T05:04:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.928+0000 7f4eb17fa700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 0x7f4ea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.928+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ea8040fb0 con 0x7f4eb40f5410 2023-11-15T05:04:02.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.928+0000 7f4eb3fff700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 0x7f4ea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:02.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.928+0000 7f4eb3fff700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 0x7f4ea4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4ea00096a0 tx=0x7f4ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:02.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:02.932+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ea80095f0 con 0x7f4eb40f5410 2023-11-15T05:04:03.140 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.139+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4eb410a340 con 0x7f4eb40f5410 2023-11-15T05:04:03.141 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.140+0000 7f4eb17fa700 1 -- 172.21.15.154:0/880774564 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ea801db70 con 0x7f4eb40f5410 2023-11-15T05:04:03.141 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:03.141 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:03.143 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 msgr2=0x7f4ea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:03.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 0x7f4ea4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f4ea00096a0 tx=0x7f4ea0006b40).stop 2023-11-15T05:04:03.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 msgr2=0x7f4eb41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:03.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4ea8013510 tx=0x7f4ea800b040).stop 2023-11-15T05:04:03.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 shutdown_connections 2023-11-15T05:04:03.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ea402ff60 0x7f4ea4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:03.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 --2- 172.21.15.154:0/880774564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40f5410 0x7f4eb41093f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:03.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 >> 172.21.15.154:0/880774564 conn(0x7f4eb40f0b70 msgr2=0x7f4eb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:03.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 shutdown_connections 2023-11-15T05:04:03.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:03.142+0000 7f4ebb205700 1 -- 172.21.15.154:0/880774564 wait complete. 2023-11-15T05:04:03.145 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:03.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:02.820597+0000 mgr.smithi086.hkfamo (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:04.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:03.141184+0000 mon.smithi086 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.154:0/880774564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:04.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:04.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:04.820905+0000 mgr.smithi086.hkfamo (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:05.701 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- 172.21.15.154:0/2534083979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 msgr2=0x7f137c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:05.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 --2- 172.21.15.154:0/2534083979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c0f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f1378009fb0 tx=0x7f1378009b30).stop 2023-11-15T05:04:05.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- 172.21.15.154:0/2534083979 shutdown_connections 2023-11-15T05:04:05.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 --2- 172.21.15.154:0/2534083979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c0f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:05.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- 172.21.15.154:0/2534083979 >> 172.21.15.154:0/2534083979 conn(0x7f137c0f1c70 msgr2=0x7f137c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:05.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- 172.21.15.154:0/2534083979 shutdown_connections 2023-11-15T05:04:05.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- 172.21.15.154:0/2534083979 wait complete. 2023-11-15T05:04:05.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 Processor -- start 2023-11-15T05:04:05.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.700+0000 7f1383c24700 1 -- start start 2023-11-15T05:04:05.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f1383c24700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:05.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f1383c24700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c106bc0 con 0x7f137c0f6510 2023-11-15T05:04:05.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13819c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:05.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13819c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45884/0 (socket says 172.21.15.154:45884) 2023-11-15T05:04:05.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13819c0700 1 -- 172.21.15.154:0/3958010589 learned_addr learned my addr 172.21.15.154:0/3958010589 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:05.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13819c0700 1 -- 172.21.15.154:0/3958010589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378005040 con 0x7f137c0f6510 2023-11-15T05:04:05.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13819c0700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1378013510 tx=0x7f137800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:05.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378009320 con 0x7f137c0f6510 2023-11-15T05:04:05.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c106dc0 con 0x7f137c0f6510 2023-11-15T05:04:05.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1378013f00 con 0x7f137c0f6510 2023-11-15T05:04:05.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378014d90 con 0x7f137c0f6510 2023-11-15T05:04:05.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.701+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c107250 con 0x7f137c0f6510 2023-11-15T05:04:05.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.702+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13780095f0 con 0x7f137c0f6510 2023-11-15T05:04:05.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.702+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c045ad0 con 0x7f137c0f6510 2023-11-15T05:04:05.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.702+0000 7f13727fc700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 0x7f1368032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:05.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.702+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1378040fe0 con 0x7f137c0f6510 2023-11-15T05:04:05.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.703+0000 7f13811bf700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 0x7f1368032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:05.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.703+0000 7f13811bf700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 0x7f1368032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f136c0096a0 tx=0x7f136c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:05.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.707+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f137801d7f0 con 0x7f137c0f6510 2023-11-15T05:04:05.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.913+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f137c108100 con 0x7f137c0f6510 2023-11-15T05:04:05.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.914+0000 7f13727fc700 1 -- 172.21.15.154:0/3958010589 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1378010230 con 0x7f137c0f6510 2023-11-15T05:04:05.915 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:05.915 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:05.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 msgr2=0x7f1368032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:05.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 0x7f1368032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f136c0096a0 tx=0x7f136c006b40).stop 2023-11-15T05:04:05.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 msgr2=0x7f137c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:05.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f1378013510 tx=0x7f137800b040).stop 2023-11-15T05:04:05.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 shutdown_connections 2023-11-15T05:04:05.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f136802ff60 0x7f1368032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:05.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 --2- 172.21.15.154:0/3958010589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137c0f6510 0x7f137c1066d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:05.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 >> 172.21.15.154:0/3958010589 conn(0x7f137c0f1c70 msgr2=0x7f137c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:05.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 shutdown_connections 2023-11-15T05:04:05.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:05.916+0000 7f1383c24700 1 -- 172.21.15.154:0/3958010589 wait complete. 2023-11-15T05:04:05.919 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:06.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:05.915121+0000 mon.smithi086 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.154:0/3958010589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:07.304 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:07.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:06.821255+0000 mgr.smithi086.hkfamo (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 -- 172.21.15.154:0/3725424733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 msgr2=0x7fc2800f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 --2- 172.21.15.154:0/3725424733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2800f9190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc270009fb0 tx=0x7fc270009b30).stop 2023-11-15T05:04:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 -- 172.21.15.154:0/3725424733 shutdown_connections 2023-11-15T05:04:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 --2- 172.21.15.154:0/3725424733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2800f9190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 -- 172.21.15.154:0/3725424733 >> 172.21.15.154:0/3725424733 conn(0x7fc2800f0b70 msgr2=0x7fc2800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 -- 172.21.15.154:0/3725424733 shutdown_connections 2023-11-15T05:04:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 -- 172.21.15.154:0/3725424733 wait complete. 2023-11-15T05:04:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.431+0000 7fc286610700 1 Processor -- start 2023-11-15T05:04:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc286610700 1 -- start start 2023-11-15T05:04:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc286610700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc286610700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc280106bd0 con 0x7fc2800601e0 2023-11-15T05:04:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc27ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc27ffff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45906/0 (socket says 172.21.15.154:45906) 2023-11-15T05:04:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.432+0000 7fc27ffff700 1 -- 172.21.15.154:0/339686184 learned_addr learned my addr 172.21.15.154:0/339686184 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc27ffff700 1 -- 172.21.15.154:0/339686184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270005040 con 0x7fc2800601e0 2023-11-15T05:04:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc27ffff700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc270004dd0 tx=0x7fc270009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc270018070 con 0x7fc2800601e0 2023-11-15T05:04:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc280106dd0 con 0x7fc2800601e0 2023-11-15T05:04:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc270013410 con 0x7fc2800601e0 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc27001c3c0 con 0x7fc2800601e0 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.433+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc280107260 con 0x7fc2800601e0 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.434+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc27001d420 con 0x7fc2800601e0 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.434+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc280045ad0 con 0x7fc2800601e0 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.434+0000 7fc27cff9700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 0x7fc26c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.434+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc270041330 con 0x7fc2800601e0 2023-11-15T05:04:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.434+0000 7fc27f7fe700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 0x7fc26c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.435+0000 7fc27f7fe700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 0x7fc26c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc2680096a0 tx=0x7fc268006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.439+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc27001d7e0 con 0x7fc2800601e0 2023-11-15T05:04:08.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.645+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc280108110 con 0x7fc2800601e0 2023-11-15T05:04:08.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.646+0000 7fc27cff9700 1 -- 172.21.15.154:0/339686184 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc27001b370 con 0x7fc2800601e0 2023-11-15T05:04:08.647 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:08.647 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:08.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 msgr2=0x7fc26c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 0x7fc26c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc2680096a0 tx=0x7fc268006b40).stop 2023-11-15T05:04:08.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 msgr2=0x7fc2801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:08.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc270004dd0 tx=0x7fc270009b30).stop 2023-11-15T05:04:08.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 shutdown_connections 2023-11-15T05:04:08.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc26c02ff60 0x7fc26c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 --2- 172.21.15.154:0/339686184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2800601e0 0x7fc2801066e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:08.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 >> 172.21.15.154:0/339686184 conn(0x7fc2800f0b70 msgr2=0x7fc2800f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:08.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 shutdown_connections 2023-11-15T05:04:08.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:08.647+0000 7fc286610700 1 -- 172.21.15.154:0/339686184 wait complete. 2023-11-15T05:04:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:08.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:08.646898+0000 mon.smithi086 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.154:0/339686184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:09.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:08.821580+0000 mgr.smithi086.hkfamo (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:10.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:10.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:11.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 -- 172.21.15.154:0/3508092258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 msgr2=0x7f29d80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3508092258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f7d00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f29d4009fb0 tx=0x7f29d4009b30).stop 2023-11-15T05:04:11.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 -- 172.21.15.154:0/3508092258 shutdown_connections 2023-11-15T05:04:11.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3508092258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f7d00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 -- 172.21.15.154:0/3508092258 >> 172.21.15.154:0/3508092258 conn(0x7f29d80f0120 msgr2=0x7f29d80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:11.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 -- 172.21.15.154:0/3508092258 shutdown_connections 2023-11-15T05:04:11.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.223+0000 7f29dedb9700 1 -- 172.21.15.154:0/3508092258 wait complete. 2023-11-15T05:04:11.227 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dedb9700 1 Processor -- start 2023-11-15T05:04:11.227 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dedb9700 1 -- start start 2023-11-15T05:04:11.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dedb9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:11.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dedb9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80f5630 con 0x7f29d80f78e0 2023-11-15T05:04:11.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dddb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:11.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dddb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45922/0 (socket says 172.21.15.154:45922) 2023-11-15T05:04:11.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.224+0000 7f29dddb7700 1 -- 172.21.15.154:0/3759406037 learned_addr learned my addr 172.21.15.154:0/3759406037 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:11.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29dddb7700 1 -- 172.21.15.154:0/3759406037 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d4005040 con 0x7f29d80f78e0 2023-11-15T05:04:11.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29dddb7700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f29d4009830 tx=0x7f29d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:11.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d4009320 con 0x7f29d80f78e0 2023-11-15T05:04:11.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29d40136c0 con 0x7f29d80f78e0 2023-11-15T05:04:11.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d80f57d0 con 0x7f29d80f78e0 2023-11-15T05:04:11.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d401ee90 con 0x7f29d80f78e0 2023-11-15T05:04:11.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.225+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d80f5c60 con 0x7f29d80f78e0 2023-11-15T05:04:11.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.226+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29d401e420 con 0x7f29d80f78e0 2023-11-15T05:04:11.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.226+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d8059100 con 0x7f29d80f78e0 2023-11-15T05:04:11.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.226+0000 7f29ca7fc700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 0x7f29c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:11.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.226+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29d4040fe0 con 0x7f29d80f78e0 2023-11-15T05:04:11.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.226+0000 7f29dd5b6700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 0x7f29c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:11.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.227+0000 7f29dd5b6700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 0x7f29c40323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f29cc0096a0 tx=0x7f29cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:11.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.231+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29d401c790 con 0x7f29d80f78e0 2023-11-15T05:04:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:10.821947+0000 mgr.smithi086.hkfamo (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:11.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.436+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29d80fbce0 con 0x7f29d80f78e0 2023-11-15T05:04:11.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.437+0000 7f29ca7fc700 1 -- 172.21.15.154:0/3759406037 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29d402c2d0 con 0x7f29d80f78e0 2023-11-15T05:04:11.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:11.438 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:11.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 msgr2=0x7f29c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 0x7f29c40323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f29cc0096a0 tx=0x7f29cc006b40).stop 2023-11-15T05:04:11.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 msgr2=0x7f29d80f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:11.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f29d4009830 tx=0x7f29d400b040).stop 2023-11-15T05:04:11.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 shutdown_connections 2023-11-15T05:04:11.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f29c402ff10 0x7f29c40323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 --2- 172.21.15.154:0/3759406037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29d80f78e0 0x7f29d80f71b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:11.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 >> 172.21.15.154:0/3759406037 conn(0x7f29d80f0120 msgr2=0x7f29d80f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:11.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 shutdown_connections 2023-11-15T05:04:11.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:11.438+0000 7f29dedb9700 1 -- 172.21.15.154:0/3759406037 wait complete. 2023-11-15T05:04:11.441 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:12.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:11.437622+0000 mon.smithi086 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.154:0/3759406037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:12.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:12.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:12.822328+0000 mgr.smithi086.hkfamo (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:14.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.025+0000 7f019444b700 1 -- 172.21.15.154:0/1709416440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 msgr2=0x7f018c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.025+0000 7f019444b700 1 --2- 172.21.15.154:0/1709416440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0180009fb0 tx=0x7f0180009b30).stop 2023-11-15T05:04:14.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- 172.21.15.154:0/1709416440 shutdown_connections 2023-11-15T05:04:14.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 --2- 172.21.15.154:0/1709416440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- 172.21.15.154:0/1709416440 >> 172.21.15.154:0/1709416440 conn(0x7f018c0f1c70 msgr2=0x7f018c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:14.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- 172.21.15.154:0/1709416440 shutdown_connections 2023-11-15T05:04:14.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- 172.21.15.154:0/1709416440 wait complete. 2023-11-15T05:04:14.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 Processor -- start 2023-11-15T05:04:14.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- start start 2023-11-15T05:04:14.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:14.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.026+0000 7f019444b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c106c20 con 0x7f018c0f6510 2023-11-15T05:04:14.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f01921e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:14.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f01921e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53822/0 (socket says 172.21.15.154:53822) 2023-11-15T05:04:14.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f01921e7700 1 -- 172.21.15.154:0/3484183634 learned_addr learned my addr 172.21.15.154:0/3484183634 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:14.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f01921e7700 1 -- 172.21.15.154:0/3484183634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180005040 con 0x7f018c0f6510 2023-11-15T05:04:14.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f01921e7700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0180013510 tx=0x7f018000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:14.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0180009320 con 0x7f018c0f6510 2023-11-15T05:04:14.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c106e20 con 0x7f018c0f6510 2023-11-15T05:04:14.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0180013f00 con 0x7f018c0f6510 2023-11-15T05:04:14.033 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0180014d90 con 0x7f018c0f6510 2023-11-15T05:04:14.033 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.027+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c1072b0 con 0x7f018c0f6510 2023-11-15T05:04:14.033 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.028+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01800095f0 con 0x7f018c0f6510 2023-11-15T05:04:14.033 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.028+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c045ad0 con 0x7f018c0f6510 2023-11-15T05:04:14.033 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.028+0000 7f017effd700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 0x7f0178032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:14.034 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.028+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0180041080 con 0x7f018c0f6510 2023-11-15T05:04:14.034 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.028+0000 7f01919e6700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 0x7f0178032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:14.034 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.029+0000 7f01919e6700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 0x7f0178032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f01880096a0 tx=0x7f0188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:14.035 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.033+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0180014420 con 0x7f018c0f6510 2023-11-15T05:04:14.240 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.239+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f018c0fbf60 con 0x7f018c0f6510 2023-11-15T05:04:14.240 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.240+0000 7f017effd700 1 -- 172.21.15.154:0/3484183634 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0180010230 con 0x7f018c0f6510 2023-11-15T05:04:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:14.241 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:14.242 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 msgr2=0x7f0178032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.242 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 0x7f0178032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f01880096a0 tx=0x7f0188006b40).stop 2023-11-15T05:04:14.242 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 msgr2=0x7f018c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:14.243 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0180013510 tx=0x7f018000b040).stop 2023-11-15T05:04:14.243 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 shutdown_connections 2023-11-15T05:04:14.243 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f017802ff60 0x7f0178032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.243 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 --2- 172.21.15.154:0/3484183634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0f6510 0x7f018c106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:14.243 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 >> 172.21.15.154:0/3484183634 conn(0x7f018c0f1c70 msgr2=0x7f018c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:14.244 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 shutdown_connections 2023-11-15T05:04:14.244 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:14.241+0000 7f019444b700 1 -- 172.21.15.154:0/3484183634 wait complete. 2023-11-15T05:04:14.244 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:14.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:14.240837+0000 mon.smithi086 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.154:0/3484183634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:15.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:15.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:15.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:14.822721+0000 mgr.smithi086.hkfamo (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:16.774 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- 172.21.15.154:0/3529999814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 msgr2=0x7f74440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:16.774 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 --2- 172.21.15.154:0/3529999814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f74440f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7438009fb0 tx=0x7f7438009b30).stop 2023-11-15T05:04:16.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- 172.21.15.154:0/3529999814 shutdown_connections 2023-11-15T05:04:16.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 --2- 172.21.15.154:0/3529999814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f74440f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:16.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- 172.21.15.154:0/3529999814 >> 172.21.15.154:0/3529999814 conn(0x7f74440f1c70 msgr2=0x7f74440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:16.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- 172.21.15.154:0/3529999814 shutdown_connections 2023-11-15T05:04:16.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- 172.21.15.154:0/3529999814 wait complete. 2023-11-15T05:04:16.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 Processor -- start 2023-11-15T05:04:16.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.771+0000 7f744b262700 1 -- start start 2023-11-15T05:04:16.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f744b262700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:16.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f744b262700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444106c20 con 0x7f74440f6510 2023-11-15T05:04:16.777 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f7448ffe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:16.777 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f7448ffe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53850/0 (socket says 172.21.15.154:53850) 2023-11-15T05:04:16.777 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f7448ffe700 1 -- 172.21.15.154:0/584642624 learned_addr learned my addr 172.21.15.154:0/584642624 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:16.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f7448ffe700 1 -- 172.21.15.154:0/584642624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7438005040 con 0x7f74440f6510 2023-11-15T05:04:16.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.772+0000 7f7448ffe700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7438013510 tx=0x7f743800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:16.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7438009320 con 0x7f74440f6510 2023-11-15T05:04:16.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7438013f00 con 0x7f74440f6510 2023-11-15T05:04:16.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7444106e20 con 0x7f74440f6510 2023-11-15T05:04:16.779 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7438014d90 con 0x7f74440f6510 2023-11-15T05:04:16.779 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74441072b0 con 0x7f74440f6510 2023-11-15T05:04:16.780 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.773+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74380095f0 con 0x7f74440f6510 2023-11-15T05:04:16.780 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.774+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7444045ad0 con 0x7f74440f6510 2023-11-15T05:04:16.780 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.774+0000 7f74417fa700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:16.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.774+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7438041000 con 0x7f74440f6510 2023-11-15T05:04:16.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.774+0000 7f7443fff700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:16.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.774+0000 7f7443fff700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 0x7f7434032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f74300096a0 tx=0x7f7430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:16.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.778+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7438014420 con 0x7f74440f6510 2023-11-15T05:04:16.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.985+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74440fbf60 con 0x7f74440f6510 2023-11-15T05:04:16.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.986+0000 7f74417fa700 1 -- 172.21.15.154:0/584642624 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7438010230 con 0x7f74440f6510 2023-11-15T05:04:16.988 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:16.988 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:16.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 msgr2=0x7f7434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:16.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 0x7f7434032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f74300096a0 tx=0x7f7430006b40).stop 2023-11-15T05:04:16.989 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 msgr2=0x7f7444106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:16.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7438013510 tx=0x7f743800b040).stop 2023-11-15T05:04:16.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 shutdown_connections 2023-11-15T05:04:16.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:16.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 --2- 172.21.15.154:0/584642624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74440f6510 0x7f7444106730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:16.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 >> 172.21.15.154:0/584642624 conn(0x7f74440f1c70 msgr2=0x7f74440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:16.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.987+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 shutdown_connections 2023-11-15T05:04:16.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:16.988+0000 7f744b262700 1 -- 172.21.15.154:0/584642624 wait complete. 2023-11-15T05:04:16.992 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:16.823112+0000 mgr.smithi086.hkfamo (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:16.987939+0000 mon.smithi086 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.154:0/584642624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:18.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:18.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:18.823486+0000 mgr.smithi086.hkfamo (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:19.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- 172.21.15.154:0/2578067773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 msgr2=0x7f075c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:19.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 --2- 172.21.15.154:0/2578067773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c0f55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0758009fb0 tx=0x7f0758009b30).stop 2023-11-15T05:04:19.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- 172.21.15.154:0/2578067773 shutdown_connections 2023-11-15T05:04:19.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 --2- 172.21.15.154:0/2578067773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c0f55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:19.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- 172.21.15.154:0/2578067773 >> 172.21.15.154:0/2578067773 conn(0x7f075c0f09a0 msgr2=0x7f075c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:19.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- 172.21.15.154:0/2578067773 shutdown_connections 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- 172.21.15.154:0/2578067773 wait complete. 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 Processor -- start 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.567+0000 7f07634d4700 1 -- start start 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07634d4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07634d4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c106990 con 0x7f075c0f51a0 2023-11-15T05:04:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07624d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:19.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07624d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53860/0 (socket says 172.21.15.154:53860) 2023-11-15T05:04:19.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07624d2700 1 -- 172.21.15.154:0/1648282774 learned_addr learned my addr 172.21.15.154:0/1648282774 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:19.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07624d2700 1 -- 172.21.15.154:0/1648282774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0758005040 con 0x7f075c0f51a0 2023-11-15T05:04:19.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07624d2700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0758013510 tx=0x7f075800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:19.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0758009320 con 0x7f075c0f51a0 2023-11-15T05:04:19.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c106b90 con 0x7f075c0f51a0 2023-11-15T05:04:19.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0758013f00 con 0x7f075c0f51a0 2023-11-15T05:04:19.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0758014dc0 con 0x7f075c0f51a0 2023-11-15T05:04:19.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.568+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c107020 con 0x7f075c0f51a0 2023-11-15T05:04:19.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.569+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07580095f0 con 0x7f075c0f51a0 2023-11-15T05:04:19.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.569+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c045ad0 con 0x7f075c0f51a0 2023-11-15T05:04:19.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.569+0000 7f0752ffd700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 0x7f0748032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:19.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.569+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0758041000 con 0x7f075c0f51a0 2023-11-15T05:04:19.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.570+0000 7f0761cd1700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 0x7f0748032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:19.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.570+0000 7f0761cd1700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 0x7f0748032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f074c0096a0 tx=0x7f074c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:19.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.575+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0758014420 con 0x7f075c0f51a0 2023-11-15T05:04:19.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.781+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f075c107ed0 con 0x7f075c0f51a0 2023-11-15T05:04:19.783 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.781+0000 7f0752ffd700 1 -- 172.21.15.154:0/1648282774 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0758010230 con 0x7f075c0f51a0 2023-11-15T05:04:19.784 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:19.784 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:19.786 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 msgr2=0x7f0748032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:19.787 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 0x7f0748032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f074c0096a0 tx=0x7f074c006b40).stop 2023-11-15T05:04:19.787 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 msgr2=0x7f075c1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:19.787 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0758013510 tx=0x7f075800b040).stop 2023-11-15T05:04:19.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 shutdown_connections 2023-11-15T05:04:19.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f074802ff60 0x7f0748032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:19.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 --2- 172.21.15.154:0/1648282774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f075c0f51a0 0x7f075c1064a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:19.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 >> 172.21.15.154:0/1648282774 conn(0x7f075c0f09a0 msgr2=0x7f075c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:19.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 shutdown_connections 2023-11-15T05:04:19.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:19.783+0000 7f07634d4700 1 -- 172.21.15.154:0/1648282774 wait complete. 2023-11-15T05:04:19.790 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:20.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:19.782252+0000 mon.smithi086 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.154:0/1648282774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:21.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:21.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:20.823888+0000 mgr.smithi086.hkfamo (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:22.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 -- 172.21.15.154:0/1926936631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 msgr2=0x7f12f40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 --2- 172.21.15.154:0/1926936631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f40f90f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f12dc009fb0 tx=0x7f12dc009b30).stop 2023-11-15T05:04:22.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 -- 172.21.15.154:0/1926936631 shutdown_connections 2023-11-15T05:04:22.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 --2- 172.21.15.154:0/1926936631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f40f90f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 -- 172.21.15.154:0/1926936631 >> 172.21.15.154:0/1926936631 conn(0x7f12f40f0b70 msgr2=0x7f12f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 -- 172.21.15.154:0/1926936631 shutdown_connections 2023-11-15T05:04:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.349+0000 7f12f8945700 1 -- 172.21.15.154:0/1926936631 wait complete. 2023-11-15T05:04:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f8945700 1 Processor -- start 2023-11-15T05:04:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f8945700 1 -- start start 2023-11-15T05:04:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f8945700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:22.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f8945700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4106bd0 con 0x7f12f4060280 2023-11-15T05:04:22.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:22.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53868/0 (socket says 172.21.15.154:53868) 2023-11-15T05:04:22.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f259c700 1 -- 172.21.15.154:0/3381586571 learned_addr learned my addr 172.21.15.154:0/3381586571 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:22.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.350+0000 7f12f259c700 1 -- 172.21.15.154:0/3381586571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc005040 con 0x7f12f4060280 2023-11-15T05:04:22.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12f259c700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f12dc013510 tx=0x7f12dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:22.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc009320 con 0x7f12f4060280 2023-11-15T05:04:22.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f4106dd0 con 0x7f12f4060280 2023-11-15T05:04:22.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12dc013f00 con 0x7f12f4060280 2023-11-15T05:04:22.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12dc014dc0 con 0x7f12f4060280 2023-11-15T05:04:22.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.351+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4107260 con 0x7f12f4060280 2023-11-15T05:04:22.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f4045ad0 con 0x7f12f4060280 2023-11-15T05:04:22.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12dc0095f0 con 0x7f12f4060280 2023-11-15T05:04:22.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12eaffd700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:22.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12dc041020 con 0x7f12f4060280 2023-11-15T05:04:22.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12f1d9b700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:22.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.352+0000 7f12f1d9b700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 0x7f12e0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f12e40096a0 tx=0x7f12e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:22.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.356+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12dc014420 con 0x7f12f4060280 2023-11-15T05:04:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.564+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12f41080b0 con 0x7f12f4060280 2023-11-15T05:04:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.565+0000 7f12eaffd700 1 -- 172.21.15.154:0/3381586571 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12dc01c920 con 0x7f12f4060280 2023-11-15T05:04:22.566 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:22.566 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:22.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 msgr2=0x7f12e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 0x7f12e0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f12e40096a0 tx=0x7f12e4006b40).stop 2023-11-15T05:04:22.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 msgr2=0x7f12f41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:22.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f12dc013510 tx=0x7f12dc00b040).stop 2023-11-15T05:04:22.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 shutdown_connections 2023-11-15T05:04:22.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12e002ff60 0x7f12e0032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 --2- 172.21.15.154:0/3381586571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12f4060280 0x7f12f41066e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:22.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 >> 172.21.15.154:0/3381586571 conn(0x7f12f40f0b70 msgr2=0x7f12f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:22.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.566+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 shutdown_connections 2023-11-15T05:04:22.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:22.567+0000 7f12f8945700 1 -- 172.21.15.154:0/3381586571 wait complete. 2023-11-15T05:04:22.569 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:22.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:22.565816+0000 mon.smithi086 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.154:0/3381586571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:23.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:23.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:23.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:22.824288+0000 mgr.smithi086.hkfamo (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:25.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- 172.21.15.154:0/3127344805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 msgr2=0x7f0fd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 --2- 172.21.15.154:0/3127344805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd40f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f0fc4009fb0 tx=0x7f0fc4009b30).stop 2023-11-15T05:04:25.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- 172.21.15.154:0/3127344805 shutdown_connections 2023-11-15T05:04:25.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 --2- 172.21.15.154:0/3127344805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd40f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- 172.21.15.154:0/3127344805 >> 172.21.15.154:0/3127344805 conn(0x7f0fd40f0b70 msgr2=0x7f0fd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- 172.21.15.154:0/3127344805 shutdown_connections 2023-11-15T05:04:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- 172.21.15.154:0/3127344805 wait complete. 2023-11-15T05:04:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 Processor -- start 2023-11-15T05:04:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- start start 2023-11-15T05:04:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.075+0000 7f0fdb586700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4106bd0 con 0x7f0fd40f5410 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fd9322700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fd9322700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40484/0 (socket says 172.21.15.154:40484) 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fd9322700 1 -- 172.21.15.154:0/718783060 learned_addr learned my addr 172.21.15.154:0/718783060 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fd9322700 1 -- 172.21.15.154:0/718783060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4005040 con 0x7f0fd40f5410 2023-11-15T05:04:25.079 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fd9322700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0fc4013510 tx=0x7f0fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:25.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc4009320 con 0x7f0fd40f5410 2023-11-15T05:04:25.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd4106dd0 con 0x7f0fd40f5410 2023-11-15T05:04:25.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fc4013f00 con 0x7f0fd40f5410 2023-11-15T05:04:25.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.076+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc4014d90 con 0x7f0fd40f5410 2023-11-15T05:04:25.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.077+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd4107260 con 0x7f0fd40f5410 2023-11-15T05:04:25.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.077+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd4045ad0 con 0x7f0fd40f5410 2023-11-15T05:04:25.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.077+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fc40095f0 con 0x7f0fd40f5410 2023-11-15T05:04:25.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.078+0000 7f0fc9ffb700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 0x7f0fc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:25.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.078+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fc4040fe0 con 0x7f0fd40f5410 2023-11-15T05:04:25.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.078+0000 7f0fd8b21700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 0x7f0fc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:25.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.078+0000 7f0fd8b21700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 0x7f0fc0032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:25.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.082+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fc4014420 con 0x7f0fd40f5410 2023-11-15T05:04:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:24.824595+0000 mgr.smithi086.hkfamo (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:25.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.290+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fd40fae60 con 0x7f0fd40f5410 2023-11-15T05:04:25.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.291+0000 7f0fc9ffb700 1 -- 172.21.15.154:0/718783060 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0fc4010230 con 0x7f0fd40f5410 2023-11-15T05:04:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:25.293 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:25.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.292+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 msgr2=0x7f0fc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 0x7f0fc0032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f0fd00096a0 tx=0x7f0fd0006b40).stop 2023-11-15T05:04:25.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 msgr2=0x7f0fd41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:25.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0fc4013510 tx=0x7f0fc400b040).stop 2023-11-15T05:04:25.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 shutdown_connections 2023-11-15T05:04:25.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0fc002ff60 0x7f0fc0032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 --2- 172.21.15.154:0/718783060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd40f5410 0x7f0fd41066e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:25.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 >> 172.21.15.154:0/718783060 conn(0x7f0fd40f0b70 msgr2=0x7f0fd405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:25.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 shutdown_connections 2023-11-15T05:04:25.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:25.293+0000 7f0fdb586700 1 -- 172.21.15.154:0/718783060 wait complete. 2023-11-15T05:04:25.296 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:24.926941+0000 mon.smithi086 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:04:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:24.934801+0000 mon.smithi086 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:04:26.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:25.291937+0000 mon.smithi086 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.154:0/718783060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:26.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:26.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:26.824910+0000 mgr.smithi086.hkfamo (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:27.894 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- 172.21.15.154:0/3066062138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 msgr2=0x7f13f00f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:27.894 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 --2- 172.21.15.154:0/3066062138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f00f8fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f13e8009960 tx=0x7f13e8004d10).stop 2023-11-15T05:04:27.894 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- 172.21.15.154:0/3066062138 shutdown_connections 2023-11-15T05:04:27.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 --2- 172.21.15.154:0/3066062138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f00f8fc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:27.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- 172.21.15.154:0/3066062138 >> 172.21.15.154:0/3066062138 conn(0x7f13f00f09a0 msgr2=0x7f13f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:27.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- 172.21.15.154:0/3066062138 shutdown_connections 2023-11-15T05:04:27.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- 172.21.15.154:0/3066062138 wait complete. 2023-11-15T05:04:27.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 Processor -- start 2023-11-15T05:04:27.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.893+0000 7f13f4e24700 1 -- start start 2023-11-15T05:04:27.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13f4e24700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:27.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13f4e24700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f0106a10 con 0x7f13f0060140 2023-11-15T05:04:27.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13ef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:27.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13ef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40504/0 (socket says 172.21.15.154:40504) 2023-11-15T05:04:27.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13ef7fe700 1 -- 172.21.15.154:0/257376995 learned_addr learned my addr 172.21.15.154:0/257376995 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:27.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13ef7fe700 1 -- 172.21.15.154:0/257376995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e8009640 con 0x7f13f0060140 2023-11-15T05:04:27.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13ef7fe700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f13e8004b70 tx=0x7f13e8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:27.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13e8018070 con 0x7f13f0060140 2023-11-15T05:04:27.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.894+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f0106c10 con 0x7f13f0060140 2023-11-15T05:04:27.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.895+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f01070a0 con 0x7f13f0060140 2023-11-15T05:04:27.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.895+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13e800b870 con 0x7f13f0060140 2023-11-15T05:04:27.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.895+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13e801c560 con 0x7f13f0060140 2023-11-15T05:04:27.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.895+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13e8013630 con 0x7f13f0060140 2023-11-15T05:04:27.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.895+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d4004fa0 con 0x7f13f0060140 2023-11-15T05:04:27.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.896+0000 7f13e7fff700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 0x7f13dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:27.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.896+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13e80414e0 con 0x7f13f0060140 2023-11-15T05:04:27.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.896+0000 7f13eeffd700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 0x7f13dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:27.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.896+0000 7f13eeffd700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 0x7f13dc032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f13e00096a0 tx=0x7f13e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:27.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:27.900+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13e8013a30 con 0x7f13f0060140 2023-11-15T05:04:28.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.107+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13d4005670 con 0x7f13f0060140 2023-11-15T05:04:28.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.108+0000 7f13e7fff700 1 -- 172.21.15.154:0/257376995 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13e8011b10 con 0x7f13f0060140 2023-11-15T05:04:28.109 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:28.109 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:28.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 msgr2=0x7f13dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:28.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 0x7f13dc032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f13e00096a0 tx=0x7f13e0006b40).stop 2023-11-15T05:04:28.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 msgr2=0x7f13f0106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:28.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f13e8004b70 tx=0x7f13e8004d10).stop 2023-11-15T05:04:28.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 shutdown_connections 2023-11-15T05:04:28.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f13dc02ff60 0x7f13dc032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:28.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 --2- 172.21.15.154:0/257376995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13f0060140 0x7f13f0106520 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:28.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.109+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 >> 172.21.15.154:0/257376995 conn(0x7f13f00f09a0 msgr2=0x7f13f00f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:28.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.110+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 shutdown_connections 2023-11-15T05:04:28.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:28.110+0000 7f13f4e24700 1 -- 172.21.15.154:0/257376995 wait complete. 2023-11-15T05:04:28.113 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:28.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:28.109053+0000 mon.smithi086 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.154:0/257376995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:29.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:29.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:29.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:28.825136+0000 mgr.smithi086.hkfamo (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:30.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 -- 172.21.15.154:0/1016779058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 msgr2=0x7f99180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:30.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 --2- 172.21.15.154:0/1016779058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f990c009fb0 tx=0x7f990c009b30).stop 2023-11-15T05:04:30.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 -- 172.21.15.154:0/1016779058 shutdown_connections 2023-11-15T05:04:30.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 --2- 172.21.15.154:0/1016779058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:30.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 -- 172.21.15.154:0/1016779058 >> 172.21.15.154:0/1016779058 conn(0x7f99180f0b70 msgr2=0x7f99180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:30.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 -- 172.21.15.154:0/1016779058 shutdown_connections 2023-11-15T05:04:30.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.637+0000 7f991ebf0700 1 -- 172.21.15.154:0/1016779058 wait complete. 2023-11-15T05:04:30.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991ebf0700 1 Processor -- start 2023-11-15T05:04:30.640 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991ebf0700 1 -- start start 2023-11-15T05:04:30.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991ebf0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:30.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991ebf0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99180fb820 con 0x7f99180f5410 2023-11-15T05:04:30.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991c98c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:30.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991c98c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40520/0 (socket says 172.21.15.154:40520) 2023-11-15T05:04:30.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.638+0000 7f991c98c700 1 -- 172.21.15.154:0/306613138 learned_addr learned my addr 172.21.15.154:0/306613138 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:30.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f991c98c700 1 -- 172.21.15.154:0/306613138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c005040 con 0x7f99180f5410 2023-11-15T05:04:30.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f991c98c700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f990c013510 tx=0x7f990c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:30.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f990c009320 con 0x7f99180f5410 2023-11-15T05:04:30.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99180fb9c0 con 0x7f99180f5410 2023-11-15T05:04:30.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f990c013f00 con 0x7f99180f5410 2023-11-15T05:04:30.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f990c014d90 con 0x7f99180f5410 2023-11-15T05:04:30.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.639+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99180fbe50 con 0x7f99180f5410 2023-11-15T05:04:30.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f990c0095f0 con 0x7f99180f5410 2023-11-15T05:04:30.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9918059100 con 0x7f99180f5410 2023-11-15T05:04:30.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f99157fa700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:30.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f990c041000 con 0x7f99180f5410 2023-11-15T05:04:30.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f9917fff700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:30.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.640+0000 7f9917fff700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 0x7f9908032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f99040096a0 tx=0x7f9904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:30.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.645+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f990c014420 con 0x7f99180f5410 2023-11-15T05:04:30.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.854+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99180fae60 con 0x7f99180f5410 2023-11-15T05:04:30.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.855+0000 7f99157fa700 1 -- 172.21.15.154:0/306613138 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f990c01d7f0 con 0x7f99180f5410 2023-11-15T05:04:30.856 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:30.856 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:30.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 msgr2=0x7f9908032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:30.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 0x7f9908032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f99040096a0 tx=0x7f9904006b40).stop 2023-11-15T05:04:30.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 msgr2=0x7f99180fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:30.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f990c013510 tx=0x7f990c00b040).stop 2023-11-15T05:04:30.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 shutdown_connections 2023-11-15T05:04:30.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f990802ff60 0x7f9908032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:30.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 --2- 172.21.15.154:0/306613138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99180f5410 0x7f99180fd3a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:30.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 >> 172.21.15.154:0/306613138 conn(0x7f99180f0b70 msgr2=0x7f991805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:30.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 shutdown_connections 2023-11-15T05:04:30.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:30.857+0000 7f991ebf0700 1 -- 172.21.15.154:0/306613138 wait complete. 2023-11-15T05:04:30.860 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:30.825544+0000 mgr.smithi086.hkfamo (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:30.855713+0000 mon.smithi086 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.154:0/306613138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:32.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:32.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:32.825897+0000 mgr.smithi086.hkfamo (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:33.507 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.506+0000 7fd329832700 1 -- 172.21.15.154:0/4000799079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 msgr2=0x7fd3240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:33.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.506+0000 7fd329832700 1 --2- 172.21.15.154:0/4000799079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fd318009fb0 tx=0x7fd318009b30).stop 2023-11-15T05:04:33.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 -- 172.21.15.154:0/4000799079 shutdown_connections 2023-11-15T05:04:33.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 --2- 172.21.15.154:0/4000799079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:33.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 -- 172.21.15.154:0/4000799079 >> 172.21.15.154:0/4000799079 conn(0x7fd3240f0b70 msgr2=0x7fd3240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:33.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 -- 172.21.15.154:0/4000799079 shutdown_connections 2023-11-15T05:04:33.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 -- 172.21.15.154:0/4000799079 wait complete. 2023-11-15T05:04:33.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 Processor -- start 2023-11-15T05:04:33.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.507+0000 7fd329832700 1 -- start start 2023-11-15T05:04:33.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd329832700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:33.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd329832700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3240fb820 con 0x7fd3240f5410 2023-11-15T05:04:33.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd322ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd322ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:32846/0 (socket says 172.21.15.154:32846) 2023-11-15T05:04:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd322ffd700 1 -- 172.21.15.154:0/76669559 learned_addr learned my addr 172.21.15.154:0/76669559 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.508+0000 7fd322ffd700 1 -- 172.21.15.154:0/76669559 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318005040 con 0x7fd3240f5410 2023-11-15T05:04:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd322ffd700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fd318013510 tx=0x7fd31800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd318009320 con 0x7fd3240f5410 2023-11-15T05:04:33.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3240fb9c0 con 0x7fd3240f5410 2023-11-15T05:04:33.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd318013f00 con 0x7fd3240f5410 2023-11-15T05:04:33.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd318014d90 con 0x7fd3240f5410 2023-11-15T05:04:33.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.509+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3240fbe50 con 0x7fd3240f5410 2023-11-15T05:04:33.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.510+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3180263a0 con 0x7fd3240f5410 2023-11-15T05:04:33.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.510+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd324059100 con 0x7fd3240f5410 2023-11-15T05:04:33.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.510+0000 7fd30b7fe700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 0x7fd30c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:33.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.510+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3180410a0 con 0x7fd3240f5410 2023-11-15T05:04:33.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.510+0000 7fd3227fc700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 0x7fd30c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:33.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.511+0000 7fd3227fc700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 0x7fd30c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd3140096a0 tx=0x7fd314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:33.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.515+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3180126f0 con 0x7fd3240f5410 2023-11-15T05:04:33.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.719+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3240fae60 con 0x7fd3240f5410 2023-11-15T05:04:33.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.720+0000 7fd30b7fe700 1 -- 172.21.15.154:0/76669559 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd318012330 con 0x7fd3240f5410 2023-11-15T05:04:33.721 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:33.721 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:33.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 msgr2=0x7fd30c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:33.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 0x7fd30c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd3140096a0 tx=0x7fd314006b40).stop 2023-11-15T05:04:33.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 msgr2=0x7fd3240fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:33.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fd318013510 tx=0x7fd31800b040).stop 2023-11-15T05:04:33.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 shutdown_connections 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd30c02ff60 0x7fd30c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 --2- 172.21.15.154:0/76669559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3240f5410 0x7fd3240fd3a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 >> 172.21.15.154:0/76669559 conn(0x7fd3240f0b70 msgr2=0x7fd32405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 shutdown_connections 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:33.721+0000 7fd329832700 1 -- 172.21.15.154:0/76669559 wait complete. 2023-11-15T05:04:33.724 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:34.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:33.720778+0000 mon.smithi086 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.154:0/76669559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:35.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:35.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:35.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:34.826251+0000 mgr.smithi086.hkfamo (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:36.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- 172.21.15.154:0/978016751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 msgr2=0x7f36c80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:36.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 --2- 172.21.15.154:0/978016751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c80f66c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f36bc009fb0 tx=0x7f36bc009b30).stop 2023-11-15T05:04:36.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- 172.21.15.154:0/978016751 shutdown_connections 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 --2- 172.21.15.154:0/978016751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c80f66c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- 172.21.15.154:0/978016751 >> 172.21.15.154:0/978016751 conn(0x7f36c80f1aa0 msgr2=0x7f36c80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- 172.21.15.154:0/978016751 shutdown_connections 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- 172.21.15.154:0/978016751 wait complete. 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 Processor -- start 2023-11-15T05:04:36.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- start start 2023-11-15T05:04:36.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:36.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.335+0000 7f36cda6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c81069a0 con 0x7f36c80f62a0 2023-11-15T05:04:36.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cca6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:36.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cca6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:32868/0 (socket says 172.21.15.154:32868) 2023-11-15T05:04:36.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cca6b700 1 -- 172.21.15.154:0/3237462417 learned_addr learned my addr 172.21.15.154:0/3237462417 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:36.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cca6b700 1 -- 172.21.15.154:0/3237462417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc005040 con 0x7f36c80f62a0 2023-11-15T05:04:36.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cca6b700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f36bc009830 tx=0x7f36bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:36.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36bc009320 con 0x7f36c80f62a0 2023-11-15T05:04:36.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c8106ba0 con 0x7f36c80f62a0 2023-11-15T05:04:36.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36bc0136c0 con 0x7f36c80f62a0 2023-11-15T05:04:36.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36bc01ee90 con 0x7f36c80f62a0 2023-11-15T05:04:36.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.336+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8107030 con 0x7f36c80f62a0 2023-11-15T05:04:36.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.337+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36bc01e420 con 0x7f36c80f62a0 2023-11-15T05:04:36.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.337+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c8045ad0 con 0x7f36c80f62a0 2023-11-15T05:04:36.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.337+0000 7f36c57fa700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 0x7f36b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:36.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.337+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36bc040fe0 con 0x7f36c80f62a0 2023-11-15T05:04:36.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.338+0000 7f36c7fff700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 0x7f36b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:36.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.338+0000 7f36c7fff700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 0x7f36b80323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f36b40096a0 tx=0x7f36b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:36.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.342+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36bc0095f0 con 0x7f36c80f62a0 2023-11-15T05:04:36.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.547+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36c8107ee0 con 0x7f36c80f62a0 2023-11-15T05:04:36.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.548+0000 7f36c57fa700 1 -- 172.21.15.154:0/3237462417 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f36bc02c2d0 con 0x7f36c80f62a0 2023-11-15T05:04:36.549 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:36.549 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:36.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 msgr2=0x7f36b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:36.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 0x7f36b80323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f36b40096a0 tx=0x7f36b4006b40).stop 2023-11-15T05:04:36.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 msgr2=0x7f36c81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:36.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f36bc009830 tx=0x7f36bc00b040).stop 2023-11-15T05:04:36.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 shutdown_connections 2023-11-15T05:04:36.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f36b802ff10 0x7f36b80323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:36.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 --2- 172.21.15.154:0/3237462417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36c80f62a0 0x7f36c81064b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:36.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 >> 172.21.15.154:0/3237462417 conn(0x7f36c80f1aa0 msgr2=0x7f36c80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:36.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 shutdown_connections 2023-11-15T05:04:36.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:36.550+0000 7f36cda6d700 1 -- 172.21.15.154:0/3237462417 wait complete. 2023-11-15T05:04:36.553 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:36.549179+0000 mon.smithi086 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.154:0/3237462417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:37.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:36.826546+0000 mgr.smithi086.hkfamo (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:37.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:37.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:39.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- 172.21.15.154:0/2144018137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 msgr2=0x7f75ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:39.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 --2- 172.21.15.154:0/2144018137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac0f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f75a0009fb0 tx=0x7f75a0009b30).stop 2023-11-15T05:04:39.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- 172.21.15.154:0/2144018137 shutdown_connections 2023-11-15T05:04:39.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 --2- 172.21.15.154:0/2144018137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac0f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:39.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- 172.21.15.154:0/2144018137 >> 172.21.15.154:0/2144018137 conn(0x7f75ac0f09a0 msgr2=0x7f75ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:39.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- 172.21.15.154:0/2144018137 shutdown_connections 2023-11-15T05:04:39.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- 172.21.15.154:0/2144018137 wait complete. 2023-11-15T05:04:39.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 Processor -- start 2023-11-15T05:04:39.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.169+0000 7f75b1c64700 1 -- start start 2023-11-15T05:04:39.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b1c64700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:39.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b1c64700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac106fc0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b0c62700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b0c62700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:32896/0 (socket says 172.21.15.154:32896) 2023-11-15T05:04:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b0c62700 1 -- 172.21.15.154:0/1955590181 learned_addr learned my addr 172.21.15.154:0/1955590181 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b0c62700 1 -- 172.21.15.154:0/1955590181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0005040 con 0x7f75ac0f51a0 2023-11-15T05:04:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b0c62700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f75a0013510 tx=0x7f75a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:39.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a0009320 con 0x7f75ac0f51a0 2023-11-15T05:04:39.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75a0013f00 con 0x7f75ac0f51a0 2023-11-15T05:04:39.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac060220 con 0x7f75ac0f51a0 2023-11-15T05:04:39.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.170+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a0014dc0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.171+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac10bac0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.171+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75a0014420 con 0x7f75ac0f51a0 2023-11-15T05:04:39.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.171+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac045ad0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.171+0000 7f75a97fa700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 0x7f759c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:39.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.171+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75a00410b0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.172+0000 7f75abfff700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 0x7f759c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:39.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.172+0000 7f75abfff700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 0x7f759c032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f75980096a0 tx=0x7f7598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:39.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.176+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75a006d2b0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:38.826864+0000 mgr.smithi086.hkfamo (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:39.383 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.383+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75ac10c280 con 0x7f75ac0f51a0 2023-11-15T05:04:39.384 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.383+0000 7f75a97fa700 1 -- 172.21.15.154:0/1955590181 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75a00095f0 con 0x7f75ac0f51a0 2023-11-15T05:04:39.384 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:39.384 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:39.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 msgr2=0x7f759c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:39.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 0x7f759c032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f75980096a0 tx=0x7f7598006b40).stop 2023-11-15T05:04:39.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 msgr2=0x7f75ac10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:39.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f75a0013510 tx=0x7f75a000b040).stop 2023-11-15T05:04:39.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 shutdown_connections 2023-11-15T05:04:39.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f759c02ffb0 0x7f759c032470 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:39.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 --2- 172.21.15.154:0/1955590181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75ac0f51a0 0x7f75ac10b5d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:39.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 >> 172.21.15.154:0/1955590181 conn(0x7f75ac0f09a0 msgr2=0x7f75ac0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:39.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 shutdown_connections 2023-11-15T05:04:39.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:39.385+0000 7f75b1c64700 1 -- 172.21.15.154:0/1955590181 wait complete. 2023-11-15T05:04:39.388 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:39.384397+0000 mon.smithi086 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.154:0/1955590181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:40.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:40.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:40.827239+0000 mgr.smithi086.hkfamo (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:41.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 -- 172.21.15.154:0/4238791516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 msgr2=0x7fe58c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:41.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 --2- 172.21.15.154:0/4238791516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c0f7d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe57c009fb0 tx=0x7fe57c009b30).stop 2023-11-15T05:04:41.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 -- 172.21.15.154:0/4238791516 shutdown_connections 2023-11-15T05:04:41.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 --2- 172.21.15.154:0/4238791516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c0f7d00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:41.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 -- 172.21.15.154:0/4238791516 >> 172.21.15.154:0/4238791516 conn(0x7fe58c0f0120 msgr2=0x7fe58c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:41.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 -- 172.21.15.154:0/4238791516 shutdown_connections 2023-11-15T05:04:41.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.973+0000 7fe58b59e700 1 -- 172.21.15.154:0/4238791516 wait complete. 2023-11-15T05:04:41.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58b59e700 1 Processor -- start 2023-11-15T05:04:41.980 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58b59e700 1 -- start start 2023-11-15T05:04:41.980 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58b59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:41.980 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58b59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c108bc0 con 0x7fe58c0f78e0 2023-11-15T05:04:41.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:41.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:32910/0 (socket says 172.21.15.154:32910) 2023-11-15T05:04:41.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.974+0000 7fe58a59c700 1 -- 172.21.15.154:0/2453139403 learned_addr learned my addr 172.21.15.154:0/2453139403 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:41.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe58a59c700 1 -- 172.21.15.154:0/2453139403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c005040 con 0x7fe58c0f78e0 2023-11-15T05:04:41.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe58a59c700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe57c013510 tx=0x7fe57c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:41.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57c009320 con 0x7fe58c0f78e0 2023-11-15T05:04:41.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c108dc0 con 0x7fe58c0f78e0 2023-11-15T05:04:41.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe57c013f00 con 0x7fe58c0f78e0 2023-11-15T05:04:41.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57c014dc0 con 0x7fe58c0f78e0 2023-11-15T05:04:41.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.975+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c109250 con 0x7fe58c0f78e0 2023-11-15T05:04:41.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c045ad0 con 0x7fe58c0f78e0 2023-11-15T05:04:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe57c0095f0 con 0x7fe58c0f78e0 2023-11-15T05:04:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe5837fe700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 0x7fe578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe57c041000 con 0x7fe58c0f78e0 2023-11-15T05:04:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe581d9b700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 0x7fe578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.976+0000 7fe581d9b700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 0x7fe578032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe5740096a0 tx=0x7fe574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:41.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:41.980+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe57c01c790 con 0x7fe58c0f78e0 2023-11-15T05:04:42.185 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.185+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe58c10a100 con 0x7fe58c0f78e0 2023-11-15T05:04:42.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.186+0000 7fe5837fe700 1 -- 172.21.15.154:0/2453139403 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe57c010230 con 0x7fe58c0f78e0 2023-11-15T05:04:42.187 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:42.187 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:42.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 msgr2=0x7fe578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:42.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 0x7fe578032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe5740096a0 tx=0x7fe574006b40).stop 2023-11-15T05:04:42.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 msgr2=0x7fe58c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:42.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe57c013510 tx=0x7fe57c00b040).stop 2023-11-15T05:04:42.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 shutdown_connections 2023-11-15T05:04:42.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe57802ff60 0x7fe578032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:42.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 --2- 172.21.15.154:0/2453139403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe58c0f78e0 0x7fe58c1086d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:42.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 >> 172.21.15.154:0/2453139403 conn(0x7fe58c0f0120 msgr2=0x7fe58c0f0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:42.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 shutdown_connections 2023-11-15T05:04:42.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:42.187+0000 7fe58b59e700 1 -- 172.21.15.154:0/2453139403 wait complete. 2023-11-15T05:04:42.190 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:42.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:42.186716+0000 mon.smithi086 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.154:0/2453139403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:43.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:43.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:43.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:42.827532+0000 mgr.smithi086.hkfamo (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:44.761 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 -- 172.21.15.154:0/4218069719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 msgr2=0x7f38fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:44.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 --2- 172.21.15.154:0/4218069719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc0f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f38ec009fb0 tx=0x7f38ec009b30).stop 2023-11-15T05:04:44.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 -- 172.21.15.154:0/4218069719 shutdown_connections 2023-11-15T05:04:44.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 --2- 172.21.15.154:0/4218069719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc0f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:44.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 -- 172.21.15.154:0/4218069719 >> 172.21.15.154:0/4218069719 conn(0x7f38fc0f0b70 msgr2=0x7f38fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:44.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 -- 172.21.15.154:0/4218069719 shutdown_connections 2023-11-15T05:04:44.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.756+0000 7f3902545700 1 -- 172.21.15.154:0/4218069719 wait complete. 2023-11-15T05:04:44.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.757+0000 7f3902545700 1 Processor -- start 2023-11-15T05:04:44.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.757+0000 7f3902545700 1 -- start start 2023-11-15T05:04:44.764 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f3902545700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:44.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f3902545700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc060240 con 0x7f38fc0f5410 2023-11-15T05:04:44.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f38fbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:44.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f38fbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:50902/0 (socket says 172.21.15.154:50902) 2023-11-15T05:04:44.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f38fbfff700 1 -- 172.21.15.154:0/3863950807 learned_addr learned my addr 172.21.15.154:0/3863950807 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:44.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f38fbfff700 1 -- 172.21.15.154:0/3863950807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec005040 con 0x7f38fc0f5410 2023-11-15T05:04:44.766 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.758+0000 7f38fbfff700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f38ec013510 tx=0x7f38ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:44.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38ec009320 con 0x7f38fc0f5410 2023-11-15T05:04:44.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc060440 con 0x7f38fc0f5410 2023-11-15T05:04:44.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38ec013f00 con 0x7f38fc0f5410 2023-11-15T05:04:44.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38ec014d90 con 0x7f38fc0f5410 2023-11-15T05:04:44.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc109ae0 con 0x7f38fc0f5410 2023-11-15T05:04:44.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.759+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38ec0095f0 con 0x7f38fc0f5410 2023-11-15T05:04:44.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.760+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc045ad0 con 0x7f38fc0f5410 2023-11-15T05:04:44.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.760+0000 7f38f8ff9700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 0x7f38e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:44.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.760+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38ec0410a0 con 0x7f38fc0f5410 2023-11-15T05:04:44.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.760+0000 7f38fb7fe700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 0x7f38e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:44.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.760+0000 7f38fb7fe700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 0x7f38e4032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f38f00096a0 tx=0x7f38f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:44.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.764+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38ec06d2b0 con 0x7f38fc0f5410 2023-11-15T05:04:44.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.973+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38fc0fae60 con 0x7f38fc0f5410 2023-11-15T05:04:44.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.974+0000 7f38f8ff9700 1 -- 172.21.15.154:0/3863950807 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38ec01d7f0 con 0x7f38fc0f5410 2023-11-15T05:04:44.974 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:44.975 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:44.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 msgr2=0x7f38e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 0x7f38e4032470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f38f00096a0 tx=0x7f38f0006b40).stop 2023-11-15T05:04:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 msgr2=0x7f38fc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f38ec013510 tx=0x7f38ec00b040).stop 2023-11-15T05:04:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 shutdown_connections 2023-11-15T05:04:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f38e402ffb0 0x7f38e4032470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:44.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 --2- 172.21.15.154:0/3863950807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38fc0f5410 0x7f38fc109430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:44.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 >> 172.21.15.154:0/3863950807 conn(0x7f38fc0f0b70 msgr2=0x7f38fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:44.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 shutdown_connections 2023-11-15T05:04:44.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:44.975+0000 7f3902545700 1 -- 172.21.15.154:0/3863950807 wait complete. 2023-11-15T05:04:44.979 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:44.827808+0000 mgr.smithi086.hkfamo (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:46.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:44.974476+0000 mon.smithi086 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.154:0/3863950807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:46.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:46.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:47.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 -- 172.21.15.154:0/4122252523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 msgr2=0x7f8f800f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 --2- 172.21.15.154:0/4122252523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f800f9150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8f70009fb0 tx=0x7f8f70009b30).stop 2023-11-15T05:04:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 -- 172.21.15.154:0/4122252523 shutdown_connections 2023-11-15T05:04:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 --2- 172.21.15.154:0/4122252523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f800f9150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 -- 172.21.15.154:0/4122252523 >> 172.21.15.154:0/4122252523 conn(0x7f8f800f0b70 msgr2=0x7f8f800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 -- 172.21.15.154:0/4122252523 shutdown_connections 2023-11-15T05:04:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.571+0000 7f8f8726b700 1 -- 172.21.15.154:0/4122252523 wait complete. 2023-11-15T05:04:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f8726b700 1 Processor -- start 2023-11-15T05:04:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f8726b700 1 -- start start 2023-11-15T05:04:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f8726b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:47.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f8726b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8010d250 con 0x7f8f80060220 2023-11-15T05:04:47.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f85007700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:47.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f85007700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:50912/0 (socket says 172.21.15.154:50912) 2023-11-15T05:04:47.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.572+0000 7f8f85007700 1 -- 172.21.15.154:0/1644674076 learned_addr learned my addr 172.21.15.154:0/1644674076 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:47.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f85007700 1 -- 172.21.15.154:0/1644674076 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f70005040 con 0x7f8f80060220 2023-11-15T05:04:47.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f85007700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f8f70013510 tx=0x7f8f7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:47.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f70009320 con 0x7f8f80060220 2023-11-15T05:04:47.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8010d450 con 0x7f8f80060220 2023-11-15T05:04:47.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f70013f00 con 0x7f8f80060220 2023-11-15T05:04:47.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f70014dc0 con 0x7f8f80060220 2023-11-15T05:04:47.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.573+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8010e060 con 0x7f8f80060220 2023-11-15T05:04:47.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.574+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f700095f0 con 0x7f8f80060220 2023-11-15T05:04:47.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.574+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8010d8e0 con 0x7f8f80060220 2023-11-15T05:04:47.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.574+0000 7f8f75ffb700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 0x7f8f6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:47.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.574+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f70041040 con 0x7f8f80060220 2023-11-15T05:04:47.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.575+0000 7f8f84806700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 0x7f8f6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:47.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.575+0000 7f8f84806700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 0x7f8f6c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8f7c0096a0 tx=0x7f8f7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:47.581 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.579+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f8010d8e0 con 0x7f8f80060220 2023-11-15T05:04:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:46.828049+0000 mgr.smithi086.hkfamo (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:47.782 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.781+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f8010d8e0 con 0x7f8f80060220 2023-11-15T05:04:47.782 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.782+0000 7f8f75ffb700 1 -- 172.21.15.154:0/1644674076 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f8010d8e0 con 0x7f8f80060220 2023-11-15T05:04:47.783 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:47.783 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:47.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 msgr2=0x7f8f6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:47.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 0x7f8f6c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8f7c0096a0 tx=0x7f8f7c006b40).stop 2023-11-15T05:04:47.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 msgr2=0x7f8f8010cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:47.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f8f70013510 tx=0x7f8f7000b040).stop 2023-11-15T05:04:47.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 shutdown_connections 2023-11-15T05:04:47.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6c02ff60 0x7f8f6c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:47.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 --2- 172.21.15.154:0/1644674076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f80060220 0x7f8f8010cd60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:47.786 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 >> 172.21.15.154:0/1644674076 conn(0x7f8f800f0b70 msgr2=0x7f8f800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:47.786 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.783+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 shutdown_connections 2023-11-15T05:04:47.786 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:47.784+0000 7f8f8726b700 1 -- 172.21.15.154:0/1644674076 wait complete. 2023-11-15T05:04:47.786 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:48.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:47.782765+0000 mon.smithi086 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.154:0/1644674076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:49.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:49.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:49.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:48.828385+0000 mgr.smithi086.hkfamo (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:50.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- 172.21.15.154:0/2330223459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 msgr2=0x7f12cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:50.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 --2- 172.21.15.154:0/2330223459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f12bc009fb0 tx=0x7f12bc009b30).stop 2023-11-15T05:04:50.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- 172.21.15.154:0/2330223459 shutdown_connections 2023-11-15T05:04:50.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 --2- 172.21.15.154:0/2330223459 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:50.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- 172.21.15.154:0/2330223459 >> 172.21.15.154:0/2330223459 conn(0x7f12cc0f09a0 msgr2=0x7f12cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:50.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- 172.21.15.154:0/2330223459 shutdown_connections 2023-11-15T05:04:50.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- 172.21.15.154:0/2330223459 wait complete. 2023-11-15T05:04:50.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 Processor -- start 2023-11-15T05:04:50.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.354+0000 7f12cb59e700 1 -- start start 2023-11-15T05:04:50.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12cb59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:50.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12cb59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc0fb5a0 con 0x7f12cc0f51a0 2023-11-15T05:04:50.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12ca59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:50.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12ca59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:50934/0 (socket says 172.21.15.154:50934) 2023-11-15T05:04:50.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12ca59c700 1 -- 172.21.15.154:0/4226390807 learned_addr learned my addr 172.21.15.154:0/4226390807 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:50.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12ca59c700 1 -- 172.21.15.154:0/4226390807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc005040 con 0x7f12cc0f51a0 2023-11-15T05:04:50.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12ca59c700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f12bc013510 tx=0x7f12bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:50.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12bc009320 con 0x7f12cc0f51a0 2023-11-15T05:04:50.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12cc0fb740 con 0x7f12cc0f51a0 2023-11-15T05:04:50.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12bc013f00 con 0x7f12cc0f51a0 2023-11-15T05:04:50.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12bc014dc0 con 0x7f12cc0f51a0 2023-11-15T05:04:50.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.355+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12cc0fbbd0 con 0x7f12cc0f51a0 2023-11-15T05:04:50.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.356+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12bc0095f0 con 0x7f12cc0f51a0 2023-11-15T05:04:50.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.356+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12cc059100 con 0x7f12cc0f51a0 2023-11-15T05:04:50.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.356+0000 7f12baffd700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 0x7f12b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:50.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.356+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12bc041020 con 0x7f12cc0f51a0 2023-11-15T05:04:50.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.357+0000 7f12c9d9b700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 0x7f12b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:50.364 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.357+0000 7f12c9d9b700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 0x7f12b4032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f12c00096a0 tx=0x7f12c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:50.364 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.361+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12bc014420 con 0x7f12cc0f51a0 2023-11-15T05:04:50.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.566+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12cc045ad0 con 0x7f12cc0f51a0 2023-11-15T05:04:50.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.567+0000 7f12baffd700 1 -- 172.21.15.154:0/4226390807 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12bc01c920 con 0x7f12cc0f51a0 2023-11-15T05:04:50.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:50.568 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:50.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 msgr2=0x7f12b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:50.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 0x7f12b4032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f12c00096a0 tx=0x7f12c0006b40).stop 2023-11-15T05:04:50.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 msgr2=0x7f12cc0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:50.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f12bc013510 tx=0x7f12bc00b040).stop 2023-11-15T05:04:50.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 shutdown_connections 2023-11-15T05:04:50.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f12b402ff60 0x7f12b4032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:50.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 --2- 172.21.15.154:0/4226390807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12cc0f51a0 0x7f12cc0fd120 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:50.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 >> 172.21.15.154:0/4226390807 conn(0x7f12cc0f09a0 msgr2=0x7f12cc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:50.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.568+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 shutdown_connections 2023-11-15T05:04:50.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:50.569+0000 7f12cb59e700 1 -- 172.21.15.154:0/4226390807 wait complete. 2023-11-15T05:04:50.571 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:50.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:50.567895+0000 mon.smithi086 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.154:0/4226390807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:51.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:51.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:51.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:50.828749+0000 mgr.smithi086.hkfamo (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:53.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 -- 172.21.15.154:0/235067812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 msgr2=0x7f5e380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:53.171 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 --2- 172.21.15.154:0/235067812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e380f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5e20009fb0 tx=0x7f5e20009b30).stop 2023-11-15T05:04:53.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 -- 172.21.15.154:0/235067812 shutdown_connections 2023-11-15T05:04:53.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 --2- 172.21.15.154:0/235067812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e380f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:53.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 -- 172.21.15.154:0/235067812 >> 172.21.15.154:0/235067812 conn(0x7f5e380f0b70 msgr2=0x7f5e380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:53.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 -- 172.21.15.154:0/235067812 shutdown_connections 2023-11-15T05:04:53.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.169+0000 7f5e3d044700 1 -- 172.21.15.154:0/235067812 wait complete. 2023-11-15T05:04:53.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e3d044700 1 Processor -- start 2023-11-15T05:04:53.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e3d044700 1 -- start start 2023-11-15T05:04:53.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e3d044700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:53.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e3d044700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e38105b20 con 0x7f5e380f5410 2023-11-15T05:04:53.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e36d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:53.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e36d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:50948/0 (socket says 172.21.15.154:50948) 2023-11-15T05:04:53.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.170+0000 7f5e36d9d700 1 -- 172.21.15.154:0/581959329 learned_addr learned my addr 172.21.15.154:0/581959329 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:53.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e36d9d700 1 -- 172.21.15.154:0/581959329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e20005040 con 0x7f5e380f5410 2023-11-15T05:04:53.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e36d9d700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f5e20013510 tx=0x7f5e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:53.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e20009320 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e38105d20 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e20013f00 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e20014d90 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.171+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e381061b0 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.172+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e20014420 con 0x7f5e380f5410 2023-11-15T05:04:53.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.172+0000 7f5e2f7fe700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 0x7f5e24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:53.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.172+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e38045ad0 con 0x7f5e380f5410 2023-11-15T05:04:53.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.172+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e20040fd0 con 0x7f5e380f5410 2023-11-15T05:04:53.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.172+0000 7f5e3659c700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 0x7f5e24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:53.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.173+0000 7f5e3659c700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 0x7f5e24032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5e280096a0 tx=0x7f5e28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:53.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.177+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e2001dd50 con 0x7f5e380f5410 2023-11-15T05:04:53.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:52.829027+0000 mgr.smithi086.hkfamo (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:53.383 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.381+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e380faf50 con 0x7f5e380f5410 2023-11-15T05:04:53.383 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.382+0000 7f5e2f7fe700 1 -- 172.21.15.154:0/581959329 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e20012330 con 0x7f5e380f5410 2023-11-15T05:04:53.384 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:53.384 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:53.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 msgr2=0x7f5e24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:53.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 0x7f5e24032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5e280096a0 tx=0x7f5e28006b40).stop 2023-11-15T05:04:53.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 msgr2=0x7f5e38105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:53.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f5e20013510 tx=0x7f5e2000b040).stop 2023-11-15T05:04:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 shutdown_connections 2023-11-15T05:04:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e2402ffb0 0x7f5e24032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 --2- 172.21.15.154:0/581959329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e380f5410 0x7f5e38105630 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 >> 172.21.15.154:0/581959329 conn(0x7f5e380f0b70 msgr2=0x7f5e380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 shutdown_connections 2023-11-15T05:04:53.388 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:53.384+0000 7f5e3d044700 1 -- 172.21.15.154:0/581959329 wait complete. 2023-11-15T05:04:53.388 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:54.241 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:53.383967+0000 mon.smithi086 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.154:0/581959329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:54.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:54.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:54.829335+0000 mgr.smithi086.hkfamo (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:55.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.968+0000 7f34cf59e700 1 -- 172.21.15.154:0/4136472971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 msgr2=0x7f34d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:55.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4136472971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d00f66c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f34c0009960 tx=0x7f34c0004d10).stop 2023-11-15T05:04:55.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- 172.21.15.154:0/4136472971 shutdown_connections 2023-11-15T05:04:55.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4136472971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d00f66c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:55.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- 172.21.15.154:0/4136472971 >> 172.21.15.154:0/4136472971 conn(0x7f34d00f1aa0 msgr2=0x7f34d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:55.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- 172.21.15.154:0/4136472971 shutdown_connections 2023-11-15T05:04:55.973 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- 172.21.15.154:0/4136472971 wait complete. 2023-11-15T05:04:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 Processor -- start 2023-11-15T05:04:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- start start 2023-11-15T05:04:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.969+0000 7f34cf59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d0069e20 con 0x7f34d00f62a0 2023-11-15T05:04:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34ce59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34ce59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:41276/0 (socket says 172.21.15.154:41276) 2023-11-15T05:04:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34ce59c700 1 -- 172.21.15.154:0/4254337014 learned_addr learned my addr 172.21.15.154:0/4254337014 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34ce59c700 1 -- 172.21.15.154:0/4254337014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c0009640 con 0x7f34d00f62a0 2023-11-15T05:04:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34ce59c700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f34c0007bd0 tx=0x7f34c0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c0018070 con 0x7f34d00f62a0 2023-11-15T05:04:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d0069fc0 con 0x7f34d00f62a0 2023-11-15T05:04:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34c000b870 con 0x7f34d00f62a0 2023-11-15T05:04:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c001d420 con 0x7f34d00f62a0 2023-11-15T05:04:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.970+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d006a450 con 0x7f34d00f62a0 2023-11-15T05:04:55.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.971+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34c0024060 con 0x7f34d00f62a0 2023-11-15T05:04:55.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.971+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d0059100 con 0x7f34d00f62a0 2023-11-15T05:04:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.971+0000 7f34c6ffd700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 0x7f34bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.971+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34c00414d0 con 0x7f34d00f62a0 2023-11-15T05:04:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.973+0000 7f34cdd9b700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 0x7f34bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.973+0000 7f34cdd9b700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 0x7f34bc032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:55.976+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34c0025750 con 0x7f34d00f62a0 2023-11-15T05:04:56.184 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.182+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34d0045ad0 con 0x7f34d00f62a0 2023-11-15T05:04:56.184 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.183+0000 7f34c6ffd700 1 -- 172.21.15.154:0/4254337014 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34c0011320 con 0x7f34d00f62a0 2023-11-15T05:04:56.185 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:56.185 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:56.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 msgr2=0x7f34bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:56.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 0x7f34bc032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).stop 2023-11-15T05:04:56.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 msgr2=0x7f34d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:56.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f34c0007bd0 tx=0x7f34c0004d10).stop 2023-11-15T05:04:56.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 shutdown_connections 2023-11-15T05:04:56.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f34bc02ffb0 0x7f34bc032470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:56.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 --2- 172.21.15.154:0/4254337014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34d00f62a0 0x7f34d006b9a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:56.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 >> 172.21.15.154:0/4254337014 conn(0x7f34d00f1aa0 msgr2=0x7f34d00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:56.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.184+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 shutdown_connections 2023-11-15T05:04:56.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:56.185+0000 7f34cf59e700 1 -- 172.21.15.154:0/4254337014 wait complete. 2023-11-15T05:04:56.188 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:56.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:56.184919+0000 mon.smithi086 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.154:0/4254337014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:04:57.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:04:57.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:04:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:56.829735+0000 mgr.smithi086.hkfamo (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:04:58.761 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- 172.21.15.154:0/710218954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 msgr2=0x7fc1800f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:58.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 --2- 172.21.15.154:0/710218954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc1800f4be0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fc170009fb0 tx=0x7fc170009b30).stop 2023-11-15T05:04:58.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- 172.21.15.154:0/710218954 shutdown_connections 2023-11-15T05:04:58.762 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 --2- 172.21.15.154:0/710218954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc1800f4be0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:58.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- 172.21.15.154:0/710218954 >> 172.21.15.154:0/710218954 conn(0x7fc1800f02e0 msgr2=0x7fc1800f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:58.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- 172.21.15.154:0/710218954 shutdown_connections 2023-11-15T05:04:58.763 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- 172.21.15.154:0/710218954 wait complete. 2023-11-15T05:04:58.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 Processor -- start 2023-11-15T05:04:58.765 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.761+0000 7fc185477700 1 -- start start 2023-11-15T05:04:58.766 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc185477700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:58.766 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc185477700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc180106bf0 con 0x7fc1800f47c0 2023-11-15T05:04:58.766 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc17effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:58.766 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc17effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:41286/0 (socket says 172.21.15.154:41286) 2023-11-15T05:04:58.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc17effd700 1 -- 172.21.15.154:0/2121101761 learned_addr learned my addr 172.21.15.154:0/2121101761 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:04:58.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc17effd700 1 -- 172.21.15.154:0/2121101761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170005040 con 0x7fc1800f47c0 2023-11-15T05:04:58.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc17effd700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc170013510 tx=0x7fc17000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:58.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc170009320 con 0x7fc1800f47c0 2023-11-15T05:04:58.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc180106df0 con 0x7fc1800f47c0 2023-11-15T05:04:58.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc170013f00 con 0x7fc1800f47c0 2023-11-15T05:04:58.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.762+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc170014d90 con 0x7fc1800f47c0 2023-11-15T05:04:58.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.763+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc180107280 con 0x7fc1800f47c0 2023-11-15T05:04:58.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.763+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc170014420 con 0x7fc1800f47c0 2023-11-15T05:04:58.768 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.763+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc180045ad0 con 0x7fc1800f47c0 2023-11-15T05:04:58.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.763+0000 7fc1677fe700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:04:58.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.763+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc170041000 con 0x7fc1800f47c0 2023-11-15T05:04:58.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.764+0000 7fc17e7fc700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:04:58.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.764+0000 7fc17e7fc700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 0x7fc168032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc1740096a0 tx=0x7fc174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:04:58.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.768+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1700095f0 con 0x7fc1800f47c0 2023-11-15T05:04:58.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.974+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc180108130 con 0x7fc1800f47c0 2023-11-15T05:04:58.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.974+0000 7fc1677fe700 1 -- 172.21.15.154:0/2121101761 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc17001db70 con 0x7fc1800f47c0 2023-11-15T05:04:58.976 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:04:58.976 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:04:58.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 msgr2=0x7fc168032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:58.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 0x7fc168032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc1740096a0 tx=0x7fc174006b40).stop 2023-11-15T05:04:58.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 msgr2=0x7fc180106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:04:58.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc170013510 tx=0x7fc17000b040).stop 2023-11-15T05:04:58.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 shutdown_connections 2023-11-15T05:04:58.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc16802ff60 0x7fc168032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:58.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 --2- 172.21.15.154:0/2121101761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1800f47c0 0x7fc180106700 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:04:58.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 >> 172.21.15.154:0/2121101761 conn(0x7fc1800f02e0 msgr2=0x7fc1800f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:04:58.978 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 shutdown_connections 2023-11-15T05:04:58.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:04:58.976+0000 7fc185477700 1 -- 172.21.15.154:0/2121101761 wait complete. 2023-11-15T05:04:58.979 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:04:59.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:04:58.830071+0000 mgr.smithi086.hkfamo (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:00.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:04:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:04:58.975502+0000 mon.smithi086 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.154:0/2121101761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:00.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:00.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:00.830410+0000 mgr.smithi086.hkfamo (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:01.540 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 -- 172.21.15.154:0/94423151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 msgr2=0x7f42a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:01.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 --2- 172.21.15.154:0/94423151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a00f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f4290009fb0 tx=0x7f4290009b30).stop 2023-11-15T05:05:01.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 -- 172.21.15.154:0/94423151 shutdown_connections 2023-11-15T05:05:01.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 --2- 172.21.15.154:0/94423151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a00f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:01.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 -- 172.21.15.154:0/94423151 >> 172.21.15.154:0/94423151 conn(0x7f42a00f1c70 msgr2=0x7f42a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:01.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 -- 172.21.15.154:0/94423151 shutdown_connections 2023-11-15T05:05:01.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.539+0000 7f42a79d8700 1 -- 172.21.15.154:0/94423151 wait complete. 2023-11-15T05:05:01.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.540+0000 7f42a79d8700 1 Processor -- start 2023-11-15T05:05:01.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.540+0000 7f42a79d8700 1 -- start start 2023-11-15T05:05:01.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.540+0000 7f42a79d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.540+0000 7f42a79d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a0106c20 con 0x7f42a00f6510 2023-11-15T05:05:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a5774700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a5774700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:41294/0 (socket says 172.21.15.154:41294) 2023-11-15T05:05:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a5774700 1 -- 172.21.15.154:0/1360597057 learned_addr learned my addr 172.21.15.154:0/1360597057 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a5774700 1 -- 172.21.15.154:0/1360597057 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4290005040 con 0x7f42a00f6510 2023-11-15T05:05:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a5774700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4290013510 tx=0x7f429000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4290009320 con 0x7f42a00f6510 2023-11-15T05:05:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a0106e20 con 0x7f42a00f6510 2023-11-15T05:05:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4290013f00 con 0x7f42a00f6510 2023-11-15T05:05:01.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.541+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4290014d90 con 0x7f42a00f6510 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.542+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a01072b0 con 0x7f42a00f6510 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.542+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42900095f0 con 0x7f42a00f6510 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.542+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a0045ad0 con 0x7f42a00f6510 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.543+0000 7f4295ffb700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 0x7f428c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.543+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4290041000 con 0x7f42a00f6510 2023-11-15T05:05:01.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.543+0000 7f42a4f73700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 0x7f428c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:01.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.543+0000 7f42a4f73700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 0x7f428c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f429c0096a0 tx=0x7f429c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:01.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.547+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f429001d7f0 con 0x7f42a00f6510 2023-11-15T05:05:01.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.752+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42a00fbf60 con 0x7f42a00f6510 2023-11-15T05:05:01.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.753+0000 7f4295ffb700 1 -- 172.21.15.154:0/1360597057 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4290010230 con 0x7f42a00f6510 2023-11-15T05:05:01.754 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:01.754 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:01.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 msgr2=0x7f428c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:01.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 0x7f428c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f429c0096a0 tx=0x7f429c006b40).stop 2023-11-15T05:05:01.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 msgr2=0x7f42a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:01.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4290013510 tx=0x7f429000b040).stop 2023-11-15T05:05:01.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 shutdown_connections 2023-11-15T05:05:01.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f428c02ff60 0x7f428c032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:01.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 --2- 172.21.15.154:0/1360597057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42a00f6510 0x7f42a0106730 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:01.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 >> 172.21.15.154:0/1360597057 conn(0x7f42a00f1c70 msgr2=0x7f42a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:01.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.754+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 shutdown_connections 2023-11-15T05:05:01.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:01.755+0000 7f42a79d8700 1 -- 172.21.15.154:0/1360597057 wait complete. 2023-11-15T05:05:01.759 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:02.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:01.753965+0000 mon.smithi086 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.154:0/1360597057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:03.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:03.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:03.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:02.830773+0000 mgr.smithi086.hkfamo (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:04.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 -- 172.21.15.154:0/3941578704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 msgr2=0x7f7e380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:04.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 --2- 172.21.15.154:0/3941578704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e380f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7e30009fb0 tx=0x7f7e30009b30).stop 2023-11-15T05:05:04.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 -- 172.21.15.154:0/3941578704 shutdown_connections 2023-11-15T05:05:04.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 --2- 172.21.15.154:0/3941578704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e380f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:04.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 -- 172.21.15.154:0/3941578704 >> 172.21.15.154:0/3941578704 conn(0x7f7e380f1c70 msgr2=0x7f7e380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:04.353 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 -- 172.21.15.154:0/3941578704 shutdown_connections 2023-11-15T05:05:04.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.351+0000 7f7e3f319700 1 -- 172.21.15.154:0/3941578704 wait complete. 2023-11-15T05:05:04.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3f319700 1 Processor -- start 2023-11-15T05:05:04.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3f319700 1 -- start start 2023-11-15T05:05:04.354 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3f319700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:04.355 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3f319700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e38106c20 con 0x7f7e380f6510 2023-11-15T05:05:04.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3d0b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:04.356 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3d0b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47156/0 (socket says 172.21.15.154:47156) 2023-11-15T05:05:04.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.352+0000 7f7e3d0b5700 1 -- 172.21.15.154:0/2818096963 learned_addr learned my addr 172.21.15.154:0/2818096963 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:04.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e3d0b5700 1 -- 172.21.15.154:0/2818096963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30005040 con 0x7f7e380f6510 2023-11-15T05:05:04.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e3d0b5700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7e30013510 tx=0x7f7e3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:04.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e30009320 con 0x7f7e380f6510 2023-11-15T05:05:04.357 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e30013f00 con 0x7f7e380f6510 2023-11-15T05:05:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e38106e20 con 0x7f7e380f6510 2023-11-15T05:05:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e30014d90 con 0x7f7e380f6510 2023-11-15T05:05:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.353+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e381072b0 con 0x7f7e380f6510 2023-11-15T05:05:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.354+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e300263a0 con 0x7f7e380f6510 2023-11-15T05:05:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.354+0000 7f7e2dffb700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 0x7f7e24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:04.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.354+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e38045ad0 con 0x7f7e380f6510 2023-11-15T05:05:04.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.354+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e30040f90 con 0x7f7e380f6510 2023-11-15T05:05:04.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.354+0000 7f7e3c8b4700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 0x7f7e24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:04.359 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.355+0000 7f7e3c8b4700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 0x7f7e24032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7e280096a0 tx=0x7f7e28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:04.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.359+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e300095f0 con 0x7f7e380f6510 2023-11-15T05:05:04.564 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.563+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e380fc050 con 0x7f7e380f6510 2023-11-15T05:05:04.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.564+0000 7f7e2dffb700 1 -- 172.21.15.154:0/2818096963 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e3001db70 con 0x7f7e380f6510 2023-11-15T05:05:04.565 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:04.566 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:04.566 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 msgr2=0x7f7e24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:04.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 0x7f7e24032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7e280096a0 tx=0x7f7e28006b40).stop 2023-11-15T05:05:04.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 msgr2=0x7f7e38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:04.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7e30013510 tx=0x7f7e3000b040).stop 2023-11-15T05:05:04.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 shutdown_connections 2023-11-15T05:05:04.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7e2402ffb0 0x7f7e24032470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:04.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 --2- 172.21.15.154:0/2818096963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380f6510 0x7f7e38106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:04.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 >> 172.21.15.154:0/2818096963 conn(0x7f7e380f1c70 msgr2=0x7f7e380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:04.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 shutdown_connections 2023-11-15T05:05:04.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:04.566+0000 7f7e3f319700 1 -- 172.21.15.154:0/2818096963 wait complete. 2023-11-15T05:05:04.569 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:04.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:04.565302+0000 mon.smithi086 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.154:0/2818096963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:05.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:05.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:05.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:04.831164+0000 mgr.smithi086.hkfamo (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:07.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- 172.21.15.154:0/2647337011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 msgr2=0x7f7ab40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:07.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2647337011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab40f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f7aa8009fb0 tx=0x7f7aa8009b30).stop 2023-11-15T05:05:07.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- 172.21.15.154:0/2647337011 shutdown_connections 2023-11-15T05:05:07.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2647337011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab40f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:07.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- 172.21.15.154:0/2647337011 >> 172.21.15.154:0/2647337011 conn(0x7f7ab40f09a0 msgr2=0x7f7ab40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:07.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- 172.21.15.154:0/2647337011 shutdown_connections 2023-11-15T05:05:07.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- 172.21.15.154:0/2647337011 wait complete. 2023-11-15T05:05:07.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 Processor -- start 2023-11-15T05:05:07.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- start start 2023-11-15T05:05:07.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:07.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.089+0000 7f7ab359e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab4060240 con 0x7f7ab40f51a0 2023-11-15T05:05:07.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:07.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47174/0 (socket says 172.21.15.154:47174) 2023-11-15T05:05:07.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab259c700 1 -- 172.21.15.154:0/2288801103 learned_addr learned my addr 172.21.15.154:0/2288801103 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:07.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab259c700 1 -- 172.21.15.154:0/2288801103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa8005040 con 0x7f7ab40f51a0 2023-11-15T05:05:07.094 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab259c700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f7aa8013510 tx=0x7f7aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:07.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa8009320 con 0x7f7ab40f51a0 2023-11-15T05:05:07.096 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab4060440 con 0x7f7ab40f51a0 2023-11-15T05:05:07.097 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7aa8013f00 con 0x7f7ab40f51a0 2023-11-15T05:05:07.097 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa8014dc0 con 0x7f7ab40f51a0 2023-11-15T05:05:07.097 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.090+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab4109860 con 0x7f7ab40f51a0 2023-11-15T05:05:07.097 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.091+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7aa8014420 con 0x7f7ab40f51a0 2023-11-15T05:05:07.097 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.091+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab4045ad0 con 0x7f7ab40f51a0 2023-11-15T05:05:07.098 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.091+0000 7f7aa2ffd700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 0x7f7a9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:07.098 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.091+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7aa8041030 con 0x7f7ab40f51a0 2023-11-15T05:05:07.098 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.092+0000 7f7ab1d9b700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 0x7f7a9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:07.098 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.092+0000 7f7ab1d9b700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 0x7f7a9c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7aa40096a0 tx=0x7f7aa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:07.099 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.096+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7aa80095f0 con 0x7f7ab40f51a0 2023-11-15T05:05:07.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.305+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ab410a080 con 0x7f7ab40f51a0 2023-11-15T05:05:07.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.306+0000 7f7aa2ffd700 1 -- 172.21.15.154:0/2288801103 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7aa801cb10 con 0x7f7ab40f51a0 2023-11-15T05:05:07.307 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:07.307 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:07.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.307+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 msgr2=0x7f7a9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:07.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.307+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 0x7f7a9c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7aa40096a0 tx=0x7f7aa4006b40).stop 2023-11-15T05:05:07.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.307+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 msgr2=0x7f7ab41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:07.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.307+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f7aa8013510 tx=0x7f7aa800b040).stop 2023-11-15T05:05:07.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 shutdown_connections 2023-11-15T05:05:07.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a9c02ff60 0x7f7a9c032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:07.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 --2- 172.21.15.154:0/2288801103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ab40f51a0 0x7f7ab41091b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:07.310 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 >> 172.21.15.154:0/2288801103 conn(0x7f7ab40f09a0 msgr2=0x7f7ab40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:07.310 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 shutdown_connections 2023-11-15T05:05:07.310 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:07.308+0000 7f7ab359e700 1 -- 172.21.15.154:0/2288801103 wait complete. 2023-11-15T05:05:07.311 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:06.831499+0000 mgr.smithi086.hkfamo (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:08.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:08.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:08.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:07.306937+0000 mon.smithi086 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.154:0/2288801103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:09.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:08.831864+0000 mgr.smithi086.hkfamo (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:09.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 -- 172.21.15.154:0/4221679388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 msgr2=0x7f4f3c0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:09.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 --2- 172.21.15.154:0/4221679388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c0f5df0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4f30009fb0 tx=0x7f4f30009b30).stop 2023-11-15T05:05:09.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 -- 172.21.15.154:0/4221679388 shutdown_connections 2023-11-15T05:05:09.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 --2- 172.21.15.154:0/4221679388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c0f5df0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:09.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 -- 172.21.15.154:0/4221679388 >> 172.21.15.154:0/4221679388 conn(0x7f4f3c0f02e0 msgr2=0x7f4f3c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:09.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 -- 172.21.15.154:0/4221679388 shutdown_connections 2023-11-15T05:05:09.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.909+0000 7f4f42798700 1 -- 172.21.15.154:0/4221679388 wait complete. 2023-11-15T05:05:09.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f42798700 1 Processor -- start 2023-11-15T05:05:09.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f42798700 1 -- start start 2023-11-15T05:05:09.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f42798700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:09.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f42798700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0f5390 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f3bfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:09.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f3bfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47196/0 (socket says 172.21.15.154:47196) 2023-11-15T05:05:09.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.910+0000 7f4f3bfff700 1 -- 172.21.15.154:0/325235117 learned_addr learned my addr 172.21.15.154:0/325235117 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:09.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f3bfff700 1 -- 172.21.15.154:0/325235117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30005040 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f3bfff700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f4f30013510 tx=0x7f4f3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:09.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f30009320 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c0f5590 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f30013f00 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f30014d90 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.911+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c109a90 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.912+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f300095f0 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.912+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c059100 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.912+0000 7f4f38ff9700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 0x7f4f24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:09.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.912+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f30041000 con 0x7f4f3c0f59d0 2023-11-15T05:05:09.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.912+0000 7f4f3b7fe700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 0x7f4f24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:09.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.913+0000 7f4f3b7fe700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 0x7f4f24032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4f2c0096a0 tx=0x7f4f2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:09.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:09.917+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f30014420 con 0x7f4f3c0f59d0 2023-11-15T05:05:10.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.123+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f3c045ad0 con 0x7f4f3c0f59d0 2023-11-15T05:05:10.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.123+0000 7f4f38ff9700 1 -- 172.21.15.154:0/325235117 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f3001d7f0 con 0x7f4f3c0f59d0 2023-11-15T05:05:10.124 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:10.124 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:10.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 msgr2=0x7f4f24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:10.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 0x7f4f24032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f4f2c0096a0 tx=0x7f4f2c006b40).stop 2023-11-15T05:05:10.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 msgr2=0x7f4f3c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:10.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f4f30013510 tx=0x7f4f3000b040).stop 2023-11-15T05:05:10.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 shutdown_connections 2023-11-15T05:05:10.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f2402ff60 0x7f4f24032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:10.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 --2- 172.21.15.154:0/325235117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f3c0f59d0 0x7f4f3c1093e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:10.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 >> 172.21.15.154:0/325235117 conn(0x7f4f3c0f02e0 msgr2=0x7f4f3c0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:10.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 shutdown_connections 2023-11-15T05:05:10.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:10.125+0000 7f4f42798700 1 -- 172.21.15.154:0/325235117 wait complete. 2023-11-15T05:05:10.128 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:10.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:10.124339+0000 mon.smithi086 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.154:0/325235117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:11.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:11.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:11.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:11 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:10.832245+0000 mgr.smithi086.hkfamo (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:12.687 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- 172.21.15.154:0/3829821072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 msgr2=0x7f48b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:12.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 --2- 172.21.15.154:0/3829821072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b00f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f4898009fb0 tx=0x7f4898009b30).stop 2023-11-15T05:05:12.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- 172.21.15.154:0/3829821072 shutdown_connections 2023-11-15T05:05:12.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 --2- 172.21.15.154:0/3829821072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b00f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:12.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- 172.21.15.154:0/3829821072 >> 172.21.15.154:0/3829821072 conn(0x7f48b00f0b70 msgr2=0x7f48b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:12.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- 172.21.15.154:0/3829821072 shutdown_connections 2023-11-15T05:05:12.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- 172.21.15.154:0/3829821072 wait complete. 2023-11-15T05:05:12.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 Processor -- start 2023-11-15T05:05:12.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.685+0000 7f48b5db0700 1 -- start start 2023-11-15T05:05:12.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48b5db0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:12.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48b5db0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b0106bc0 con 0x7f48b00f5410 2023-11-15T05:05:12.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48af7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:12.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48af7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47216/0 (socket says 172.21.15.154:47216) 2023-11-15T05:05:12.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48af7fe700 1 -- 172.21.15.154:0/1698813293 learned_addr learned my addr 172.21.15.154:0/1698813293 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:12.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48af7fe700 1 -- 172.21.15.154:0/1698813293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4898005040 con 0x7f48b00f5410 2023-11-15T05:05:12.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.686+0000 7f48af7fe700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4898013510 tx=0x7f489800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:12.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4898009320 con 0x7f48b00f5410 2023-11-15T05:05:12.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b0106dc0 con 0x7f48b00f5410 2023-11-15T05:05:12.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4898013f00 con 0x7f48b00f5410 2023-11-15T05:05:12.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4898014d90 con 0x7f48b00f5410 2023-11-15T05:05:12.692 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b0107250 con 0x7f48b00f5410 2023-11-15T05:05:12.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.687+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0045ad0 con 0x7f48b00f5410 2023-11-15T05:05:12.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.688+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48980095f0 con 0x7f48b00f5410 2023-11-15T05:05:12.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.688+0000 7f48a7fff700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 0x7f489c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:12.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.688+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4898040fe0 con 0x7f48b00f5410 2023-11-15T05:05:12.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.688+0000 7f48aeffd700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 0x7f489c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:12.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.688+0000 7f48aeffd700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 0x7f489c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:12.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.692+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4898010230 con 0x7f48b00f5410 2023-11-15T05:05:12.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.897+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48b01080a0 con 0x7f48b00f5410 2023-11-15T05:05:12.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.897+0000 7f48a7fff700 1 -- 172.21.15.154:0/1698813293 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4898010230 con 0x7f48b00f5410 2023-11-15T05:05:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:12.899 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:12.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 msgr2=0x7f489c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:12.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 0x7f489c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).stop 2023-11-15T05:05:12.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 msgr2=0x7f48b01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:12.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4898013510 tx=0x7f489800b040).stop 2023-11-15T05:05:12.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 shutdown_connections 2023-11-15T05:05:12.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f489c02ff60 0x7f489c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:12.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 --2- 172.21.15.154:0/1698813293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00f5410 0x7f48b01066d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:12.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 >> 172.21.15.154:0/1698813293 conn(0x7f48b00f0b70 msgr2=0x7f48b005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:12.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 shutdown_connections 2023-11-15T05:05:12.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:12.899+0000 7f48b5db0700 1 -- 172.21.15.154:0/1698813293 wait complete. 2023-11-15T05:05:12.902 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:13.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:12.832599+0000 mgr.smithi086.hkfamo (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:14.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:12.898279+0000 mon.smithi086 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.154:0/1698813293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:14.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:14.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:15.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:14.832959+0000 mgr.smithi086.hkfamo (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:15.543 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 -- 172.21.15.154:0/1670510525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 msgr2=0x7f03040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:15.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 --2- 172.21.15.154:0/1670510525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f03040f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f02f4009fb0 tx=0x7f02f4009b30).stop 2023-11-15T05:05:15.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 -- 172.21.15.154:0/1670510525 shutdown_connections 2023-11-15T05:05:15.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 --2- 172.21.15.154:0/1670510525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f03040f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:15.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 -- 172.21.15.154:0/1670510525 >> 172.21.15.154:0/1670510525 conn(0x7f03040f0b70 msgr2=0x7f03040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 -- 172.21.15.154:0/1670510525 shutdown_connections 2023-11-15T05:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.542+0000 7f0308efc700 1 -- 172.21.15.154:0/1670510525 wait complete. 2023-11-15T05:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f0308efc700 1 Processor -- start 2023-11-15T05:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f0308efc700 1 -- start start 2023-11-15T05:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f0308efc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:15.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f0308efc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0304105b20 con 0x7f03040f5410 2023-11-15T05:05:15.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f030259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:15.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f030259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36448/0 (socket says 172.21.15.154:36448) 2023-11-15T05:05:15.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.543+0000 7f030259c700 1 -- 172.21.15.154:0/4116223413 learned_addr learned my addr 172.21.15.154:0/4116223413 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:15.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f030259c700 1 -- 172.21.15.154:0/4116223413 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4005040 con 0x7f03040f5410 2023-11-15T05:05:15.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f030259c700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f02f4013510 tx=0x7f02f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:15.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4009320 con 0x7f03040f5410 2023-11-15T05:05:15.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0304105d20 con 0x7f03040f5410 2023-11-15T05:05:15.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02f4013f00 con 0x7f03040f5410 2023-11-15T05:05:15.550 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4014d90 con 0x7f03040f5410 2023-11-15T05:05:15.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.544+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03041061b0 con 0x7f03040f5410 2023-11-15T05:05:15.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02f40095f0 con 0x7f03040f5410 2023-11-15T05:05:15.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304045ad0 con 0x7f03040f5410 2023-11-15T05:05:15.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f02faffd700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:15.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02f4041000 con 0x7f03040f5410 2023-11-15T05:05:15.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f0301d9b700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:15.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.545+0000 7f0301d9b700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 0x7f02f0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:15.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.549+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02f4014420 con 0x7f03040f5410 2023-11-15T05:05:15.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.753+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03040fae60 con 0x7f03040f5410 2023-11-15T05:05:15.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.754+0000 7f02faffd700 1 -- 172.21.15.154:0/4116223413 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02f401d7f0 con 0x7f03040f5410 2023-11-15T05:05:15.755 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:15.755 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:15.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 msgr2=0x7f02f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:15.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 0x7f02f0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).stop 2023-11-15T05:05:15.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 msgr2=0x7f0304105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:15.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f02f4013510 tx=0x7f02f400b040).stop 2023-11-15T05:05:15.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 shutdown_connections 2023-11-15T05:05:15.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:15.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 --2- 172.21.15.154:0/4116223413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03040f5410 0x7f0304105630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:15.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 >> 172.21.15.154:0/4116223413 conn(0x7f03040f0b70 msgr2=0x7f03040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:15.757 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 shutdown_connections 2023-11-15T05:05:15.758 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:15.755+0000 7f0308efc700 1 -- 172.21.15.154:0/4116223413 wait complete. 2023-11-15T05:05:15.758 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:16.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:15.754824+0000 mon.smithi086 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.154:0/4116223413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:17.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:17.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:16.833276+0000 mgr.smithi086.hkfamo (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:18.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 -- 172.21.15.154:0/972457764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 msgr2=0x7fb3640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:18.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 --2- 172.21.15.154:0/972457764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb3640f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb358009fb0 tx=0x7fb358009b30).stop 2023-11-15T05:05:18.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 -- 172.21.15.154:0/972457764 shutdown_connections 2023-11-15T05:05:18.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 --2- 172.21.15.154:0/972457764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb3640f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:18.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 -- 172.21.15.154:0/972457764 >> 172.21.15.154:0/972457764 conn(0x7fb3640f1c70 msgr2=0x7fb3640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:18.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 -- 172.21.15.154:0/972457764 shutdown_connections 2023-11-15T05:05:18.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.282+0000 7fb3699e8700 1 -- 172.21.15.154:0/972457764 wait complete. 2023-11-15T05:05:18.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.283+0000 7fb3699e8700 1 Processor -- start 2023-11-15T05:05:18.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.283+0000 7fb3699e8700 1 -- start start 2023-11-15T05:05:18.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.283+0000 7fb3699e8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:18.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.283+0000 7fb3699e8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364106c20 con 0x7fb3640f6510 2023-11-15T05:05:18.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb362ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:18.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb362ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36458/0 (socket says 172.21.15.154:36458) 2023-11-15T05:05:18.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb362ffd700 1 -- 172.21.15.154:0/666267487 learned_addr learned my addr 172.21.15.154:0/666267487 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:18.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb362ffd700 1 -- 172.21.15.154:0/666267487 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358005040 con 0x7fb3640f6510 2023-11-15T05:05:18.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb362ffd700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:18.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358009320 con 0x7fb3640f6510 2023-11-15T05:05:18.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364106e20 con 0x7fb3640f6510 2023-11-15T05:05:18.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb358013f00 con 0x7fb3640f6510 2023-11-15T05:05:18.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb358014d90 con 0x7fb3640f6510 2023-11-15T05:05:18.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.284+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3641072b0 con 0x7fb3640f6510 2023-11-15T05:05:18.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.285+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3580263a0 con 0x7fb3640f6510 2023-11-15T05:05:18.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.285+0000 7fb34b7fe700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:18.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.285+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb358041300 con 0x7fb3640f6510 2023-11-15T05:05:18.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.285+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb364059100 con 0x7fb3640f6510 2023-11-15T05:05:18.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.286+0000 7fb3627fc700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:18.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.286+0000 7fb3627fc700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 0x7fb34c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:18.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.290+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb35801cb10 con 0x7fb3640f6510 2023-11-15T05:05:18.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.497+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3640fbf60 con 0x7fb3640f6510 2023-11-15T05:05:18.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.498+0000 7fb34b7fe700 1 -- 172.21.15.154:0/666267487 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb35802b030 con 0x7fb3640f6510 2023-11-15T05:05:18.499 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:18.499 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:18.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 msgr2=0x7fb34c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:18.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 0x7fb34c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).stop 2023-11-15T05:05:18.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 msgr2=0x7fb364106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:18.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb358013510 tx=0x7fb35800b040).stop 2023-11-15T05:05:18.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 shutdown_connections 2023-11-15T05:05:18.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb34c02ff60 0x7fb34c032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:18.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 --2- 172.21.15.154:0/666267487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3640f6510 0x7fb364106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:18.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 >> 172.21.15.154:0/666267487 conn(0x7fb3640f1c70 msgr2=0x7fb3640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:18.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 shutdown_connections 2023-11-15T05:05:18.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:18.499+0000 7fb3699e8700 1 -- 172.21.15.154:0/666267487 wait complete. 2023-11-15T05:05:18.502 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:18.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:18.498950+0000 mon.smithi086 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.154:0/666267487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:19.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:19.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:19.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:18.833616+0000 mgr.smithi086.hkfamo (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:21.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 -- 172.21.15.154:0/528453685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 msgr2=0x7f6ff00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:21.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 --2- 172.21.15.154:0/528453685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff00f66c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6fd8009fb0 tx=0x7f6fd8009b30).stop 2023-11-15T05:05:21.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 -- 172.21.15.154:0/528453685 shutdown_connections 2023-11-15T05:05:21.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 --2- 172.21.15.154:0/528453685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff00f66c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:21.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 -- 172.21.15.154:0/528453685 >> 172.21.15.154:0/528453685 conn(0x7f6ff00f1aa0 msgr2=0x7f6ff00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 -- 172.21.15.154:0/528453685 shutdown_connections 2023-11-15T05:05:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 -- 172.21.15.154:0/528453685 wait complete. 2023-11-15T05:05:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.144+0000 7f6ff4822700 1 Processor -- start 2023-11-15T05:05:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6ff4822700 1 -- start start 2023-11-15T05:05:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6ff4822700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:21.148 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6ff4822700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff01069a0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.148 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6fef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:21.148 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6fef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:36478/0 (socket says 172.21.15.154:36478) 2023-11-15T05:05:21.148 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6fef7fe700 1 -- 172.21.15.154:0/2355163384 learned_addr learned my addr 172.21.15.154:0/2355163384 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:21.150 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6fef7fe700 1 -- 172.21.15.154:0/2355163384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8005040 con 0x7f6ff00f62a0 2023-11-15T05:05:21.150 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.145+0000 7f6fef7fe700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6fd8013510 tx=0x7f6fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:21.150 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.146+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8009320 con 0x7f6ff00f62a0 2023-11-15T05:05:21.150 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.146+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff0106ba0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.146+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fd8013f00 con 0x7f6ff00f62a0 2023-11-15T05:05:21.151 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.146+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8014dc0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.146+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff0107030 con 0x7f6ff00f62a0 2023-11-15T05:05:21.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fd8014420 con 0x7f6ff00f62a0 2023-11-15T05:05:21.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff0045ad0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6fe7fff700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:21.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fd80410f0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6feeffd700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:21.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.147+0000 7f6feeffd700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 0x7f6fdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6fe00096a0 tx=0x7f6fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:21.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.151+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fd80095f0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:20.833920+0000 mgr.smithi086.hkfamo (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:21.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.357+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ff0107ee0 con 0x7f6ff00f62a0 2023-11-15T05:05:21.358 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.358+0000 7f6fe7fff700 1 -- 172.21.15.154:0/2355163384 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fd801cb10 con 0x7f6ff00f62a0 2023-11-15T05:05:21.359 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:21.359 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:21.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 msgr2=0x7f6fdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:21.360 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 0x7f6fdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6fe00096a0 tx=0x7f6fe0006b40).stop 2023-11-15T05:05:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 msgr2=0x7f6ff01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6fd8013510 tx=0x7f6fd800b040).stop 2023-11-15T05:05:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 shutdown_connections 2023-11-15T05:05:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f6fdc02ff60 0x7f6fdc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 --2- 172.21.15.154:0/2355163384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff00f62a0 0x7f6ff01064b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:21.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.359+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 >> 172.21.15.154:0/2355163384 conn(0x7f6ff00f1aa0 msgr2=0x7f6ff00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:21.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.360+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 shutdown_connections 2023-11-15T05:05:21.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:21.360+0000 7f6ff4822700 1 -- 172.21.15.154:0/2355163384 wait complete. 2023-11-15T05:05:21.362 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:22.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:22.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:22.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:21.358730+0000 mon.smithi086 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.154:0/2355163384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:23.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:22.834211+0000 mgr.smithi086.hkfamo (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:23.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 -- 172.21.15.154:0/1564115343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 msgr2=0x7fe5e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:23.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/1564115343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e40f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fe5d8009fb0 tx=0x7fe5d8009b30).stop 2023-11-15T05:05:23.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 -- 172.21.15.154:0/1564115343 shutdown_connections 2023-11-15T05:05:23.905 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/1564115343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e40f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:23.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 -- 172.21.15.154:0/1564115343 >> 172.21.15.154:0/1564115343 conn(0x7fe5e40f1c70 msgr2=0x7fe5e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:23.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 -- 172.21.15.154:0/1564115343 shutdown_connections 2023-11-15T05:05:23.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.902+0000 7fe5e90ee700 1 -- 172.21.15.154:0/1564115343 wait complete. 2023-11-15T05:05:23.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.903+0000 7fe5e90ee700 1 Processor -- start 2023-11-15T05:05:23.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.903+0000 7fe5e90ee700 1 -- start start 2023-11-15T05:05:23.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.903+0000 7fe5e90ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:23.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.903+0000 7fe5e90ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e4106c20 con 0x7fe5e40f6510 2023-11-15T05:05:23.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e2d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:23.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e2d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47002/0 (socket says 172.21.15.154:47002) 2023-11-15T05:05:23.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e2d9d700 1 -- 172.21.15.154:0/469360521 learned_addr learned my addr 172.21.15.154:0/469360521 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:23.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e2d9d700 1 -- 172.21.15.154:0/469360521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8005040 con 0x7fe5e40f6510 2023-11-15T05:05:23.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e2d9d700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe5d8013510 tx=0x7fe5d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:23.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d8009320 con 0x7fe5e40f6510 2023-11-15T05:05:23.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e4106e20 con 0x7fe5e40f6510 2023-11-15T05:05:23.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5d8013f00 con 0x7fe5e40f6510 2023-11-15T05:05:23.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5d8014d90 con 0x7fe5e40f6510 2023-11-15T05:05:23.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.904+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e41072b0 con 0x7fe5e40f6510 2023-11-15T05:05:23.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.905+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5d80095f0 con 0x7fe5e40f6510 2023-11-15T05:05:23.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.905+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e4045ad0 con 0x7fe5e40f6510 2023-11-15T05:05:23.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.905+0000 7fe5cb7fe700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 0x7fe5cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:23.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.905+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5d80410c0 con 0x7fe5e40f6510 2023-11-15T05:05:23.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.905+0000 7fe5e259c700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 0x7fe5cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:23.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.906+0000 7fe5e259c700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 0x7fe5cc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe5d40096a0 tx=0x7fe5d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:23.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:23.910+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5d8014420 con 0x7fe5e40f6510 2023-11-15T05:05:24.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.117+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5e40fbf60 con 0x7fe5e40f6510 2023-11-15T05:05:24.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.118+0000 7fe5cb7fe700 1 -- 172.21.15.154:0/469360521 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5d8010230 con 0x7fe5e40f6510 2023-11-15T05:05:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:24.119 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:24.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.119+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 msgr2=0x7fe5cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:24.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.119+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 0x7fe5cc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe5d40096a0 tx=0x7fe5d4006b40).stop 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.119+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 msgr2=0x7fe5e4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.119+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe5d8013510 tx=0x7fe5d800b040).stop 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 shutdown_connections 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5cc02ff60 0x7fe5cc032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 --2- 172.21.15.154:0/469360521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5e40f6510 0x7fe5e4106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:24.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 >> 172.21.15.154:0/469360521 conn(0x7fe5e40f1c70 msgr2=0x7fe5e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:24.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 shutdown_connections 2023-11-15T05:05:24.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:24.120+0000 7fe5e90ee700 1 -- 172.21.15.154:0/469360521 wait complete. 2023-11-15T05:05:24.122 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:24.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:24.118763+0000 mon.smithi086 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.154:0/469360521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:25.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:25.508 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:25.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:24.834464+0000 mgr.smithi086.hkfamo (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:25.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:24.927923+0000 mon.smithi086 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:05:25.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:24.935662+0000 mon.smithi086 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:05:26.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 -- 172.21.15.154:0/1963601950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 msgr2=0x7ff3040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:26.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 --2- 172.21.15.154:0/1963601950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff3040f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff2f4009fb0 tx=0x7ff2f4009b30).stop 2023-11-15T05:05:26.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 -- 172.21.15.154:0/1963601950 shutdown_connections 2023-11-15T05:05:26.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 --2- 172.21.15.154:0/1963601950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff3040f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:26.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 -- 172.21.15.154:0/1963601950 >> 172.21.15.154:0/1963601950 conn(0x7ff3040f0b70 msgr2=0x7ff3040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:26.690 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 -- 172.21.15.154:0/1963601950 shutdown_connections 2023-11-15T05:05:26.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.688+0000 7ff308892700 1 -- 172.21.15.154:0/1963601950 wait complete. 2023-11-15T05:05:26.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff308892700 1 Processor -- start 2023-11-15T05:05:26.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff308892700 1 -- start start 2023-11-15T05:05:26.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff308892700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:26.691 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff308892700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff304105b20 con 0x7ff3040f5410 2023-11-15T05:05:26.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff30259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:26.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff30259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47014/0 (socket says 172.21.15.154:47014) 2023-11-15T05:05:26.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff30259c700 1 -- 172.21.15.154:0/1676942036 learned_addr learned my addr 172.21.15.154:0/1676942036 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:26.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.689+0000 7ff30259c700 1 -- 172.21.15.154:0/1676942036 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4005040 con 0x7ff3040f5410 2023-11-15T05:05:26.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff30259c700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff2f4013510 tx=0x7ff2f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:26.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f4009320 con 0x7ff3040f5410 2023-11-15T05:05:26.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff304105d20 con 0x7ff3040f5410 2023-11-15T05:05:26.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2f4013f00 con 0x7ff3040f5410 2023-11-15T05:05:26.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2f4014d90 con 0x7ff3040f5410 2023-11-15T05:05:26.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.690+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3041061b0 con 0x7ff3040f5410 2023-11-15T05:05:26.695 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff304045ad0 con 0x7ff3040f5410 2023-11-15T05:05:26.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2f40095f0 con 0x7ff3040f5410 2023-11-15T05:05:26.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff2f2ffd700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 0x7ff2ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:26.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2f4041000 con 0x7ff3040f5410 2023-11-15T05:05:26.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff301d9b700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 0x7ff2ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:26.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.691+0000 7ff301d9b700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 0x7ff2ec032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff2f80096a0 tx=0x7ff2f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:26.698 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.695+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2f4014420 con 0x7ff3040f5410 2023-11-15T05:05:26.906 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.905+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3040fae60 con 0x7ff3040f5410 2023-11-15T05:05:26.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.906+0000 7ff2f2ffd700 1 -- 172.21.15.154:0/1676942036 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2f401d7f0 con 0x7ff3040f5410 2023-11-15T05:05:26.907 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:26.907 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:26.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 msgr2=0x7ff2ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 0x7ff2ec032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff2f80096a0 tx=0x7ff2f8006b40).stop 2023-11-15T05:05:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 msgr2=0x7ff304105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff2f4013510 tx=0x7ff2f400b040).stop 2023-11-15T05:05:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 shutdown_connections 2023-11-15T05:05:26.909 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff2ec02ff60 0x7ff2ec032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:26.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 --2- 172.21.15.154:0/1676942036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3040f5410 0x7ff304105630 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:26.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 >> 172.21.15.154:0/1676942036 conn(0x7ff3040f0b70 msgr2=0x7ff3040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:26.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 shutdown_connections 2023-11-15T05:05:26.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:26.908+0000 7ff308892700 1 -- 172.21.15.154:0/1676942036 wait complete. 2023-11-15T05:05:26.911 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:26.834799+0000 mgr.smithi086.hkfamo (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:26.907137+0000 mon.smithi086 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.154:0/1676942036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:28.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:28.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:29.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:28.835103+0000 mgr.smithi086.hkfamo (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:29.540 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- 172.21.15.154:0/2841112832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 msgr2=0x7f85700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:29.540 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 --2- 172.21.15.154:0/2841112832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f85700f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f8558009fb0 tx=0x7f8558009b30).stop 2023-11-15T05:05:29.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- 172.21.15.154:0/2841112832 shutdown_connections 2023-11-15T05:05:29.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 --2- 172.21.15.154:0/2841112832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f85700f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:29.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- 172.21.15.154:0/2841112832 >> 172.21.15.154:0/2841112832 conn(0x7f85700f1c70 msgr2=0x7f85700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:29.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- 172.21.15.154:0/2841112832 shutdown_connections 2023-11-15T05:05:29.541 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- 172.21.15.154:0/2841112832 wait complete. 2023-11-15T05:05:29.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 Processor -- start 2023-11-15T05:05:29.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 -- start start 2023-11-15T05:05:29.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.539+0000 7f85749c1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:29.542 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f85749c1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8570106c20 con 0x7f85700f6510 2023-11-15T05:05:29.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f856e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:29.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f856e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47034/0 (socket says 172.21.15.154:47034) 2023-11-15T05:05:29.544 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f856e59c700 1 -- 172.21.15.154:0/3468980604 learned_addr learned my addr 172.21.15.154:0/3468980604 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:29.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f856e59c700 1 -- 172.21.15.154:0/3468980604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558005040 con 0x7f85700f6510 2023-11-15T05:05:29.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.540+0000 7f856e59c700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8558014000 tx=0x7f855800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:29.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85580095f0 con 0x7f85700f6510 2023-11-15T05:05:29.545 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8570106e20 con 0x7f85700f6510 2023-11-15T05:05:29.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8558007e70 con 0x7f85700f6510 2023-11-15T05:05:29.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8558014ea0 con 0x7f85700f6510 2023-11-15T05:05:29.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85701072b0 con 0x7f85700f6510 2023-11-15T05:05:29.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8570045ad0 con 0x7f85700f6510 2023-11-15T05:05:29.546 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.541+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8558025070 con 0x7f85700f6510 2023-11-15T05:05:29.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.542+0000 7f8566ffd700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:29.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.542+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85580409b0 con 0x7f85700f6510 2023-11-15T05:05:29.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.542+0000 7f856dd9b700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:29.547 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.542+0000 7f856dd9b700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f85600096a0 tx=0x7f8560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:29.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.546+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85580079c0 con 0x7f85700f6510 2023-11-15T05:05:29.750 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.749+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85700fbf60 con 0x7f85700f6510 2023-11-15T05:05:29.751 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.751+0000 7f8566ffd700 1 -- 172.21.15.154:0/3468980604 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f855800f300 con 0x7f85700f6510 2023-11-15T05:05:29.752 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:29.752 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:29.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 msgr2=0x7f855c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:29.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f85600096a0 tx=0x7f8560006b40).stop 2023-11-15T05:05:29.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 msgr2=0x7f8570106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:29.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8558014000 tx=0x7f855800b040).stop 2023-11-15T05:05:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 shutdown_connections 2023-11-15T05:05:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 --2- 172.21.15.154:0/3468980604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85700f6510 0x7f8570106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 >> 172.21.15.154:0/3468980604 conn(0x7f85700f1c70 msgr2=0x7f85700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 shutdown_connections 2023-11-15T05:05:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:29.752+0000 7f85749c1700 1 -- 172.21.15.154:0/3468980604 wait complete. 2023-11-15T05:05:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:30.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:29.751024+0000 mon.smithi086 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.154:0/3468980604' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:31.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:31.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:31.242 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:30.835499+0000 mgr.smithi086.hkfamo (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:32.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2452650685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 msgr2=0x7fa0bc0fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:32.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2452650685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc0fb4f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fa0ac009fb0 tx=0x7fa0ac009b30).stop 2023-11-15T05:05:32.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2452650685 shutdown_connections 2023-11-15T05:05:32.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2452650685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc0fb4f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:32.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2452650685 >> 172.21.15.154:0/2452650685 conn(0x7fa0bc0f0b70 msgr2=0x7fa0bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:32.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2452650685 shutdown_connections 2023-11-15T05:05:32.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.383+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2452650685 wait complete. 2023-11-15T05:05:32.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.384+0000 7fa0c23c5700 1 Processor -- start 2023-11-15T05:05:32.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.384+0000 7fa0c23c5700 1 -- start start 2023-11-15T05:05:32.387 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.384+0000 7fa0c23c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:32.388 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.384+0000 7fa0c23c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc108dd0 con 0x7fa0bc060280 2023-11-15T05:05:32.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0bbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:32.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0bbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47058/0 (socket says 172.21.15.154:47058) 2023-11-15T05:05:32.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0bbfff700 1 -- 172.21.15.154:0/2107988841 learned_addr learned my addr 172.21.15.154:0/2107988841 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:32.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0bbfff700 1 -- 172.21.15.154:0/2107988841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac005040 con 0x7fa0bc060280 2023-11-15T05:05:32.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0bbfff700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa0ac013510 tx=0x7fa0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:32.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac009320 con 0x7fa0bc060280 2023-11-15T05:05:32.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc108fd0 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0ac013f00 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac014dc0 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.385+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc109460 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.386+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0ac014420 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.386+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc045ad0 con 0x7fa0bc060280 2023-11-15T05:05:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.386+0000 7fa0b8ff9700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 0x7fa0a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:32.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.386+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0ac0410d0 con 0x7fa0bc060280 2023-11-15T05:05:32.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.387+0000 7fa0bb7fe700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 0x7fa0a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:32.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.387+0000 7fa0bb7fe700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 0x7fa0a8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:32.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.391+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0ac0095f0 con 0x7fa0bc060280 2023-11-15T05:05:32.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.596+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0bc10a310 con 0x7fa0bc060280 2023-11-15T05:05:32.598 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.597+0000 7fa0b8ff9700 1 -- 172.21.15.154:0/2107988841 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0ac01cb10 con 0x7fa0bc060280 2023-11-15T05:05:32.599 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:32.599 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:32.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 msgr2=0x7fa0a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:32.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 0x7fa0a8032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).stop 2023-11-15T05:05:32.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 msgr2=0x7fa0bc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:32.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa0ac013510 tx=0x7fa0ac00b040).stop 2023-11-15T05:05:32.600 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 shutdown_connections 2023-11-15T05:05:32.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0a802ff60 0x7fa0a8032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:32.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 --2- 172.21.15.154:0/2107988841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0bc060280 0x7fa0bc1088e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:32.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 >> 172.21.15.154:0/2107988841 conn(0x7fa0bc0f0b70 msgr2=0x7fa0bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:32.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 shutdown_connections 2023-11-15T05:05:32.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:32.598+0000 7fa0c23c5700 1 -- 172.21.15.154:0/2107988841 wait complete. 2023-11-15T05:05:32.602 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:32.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:32.598622+0000 mon.smithi086 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.154:0/2107988841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:33.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:32.835810+0000 mgr.smithi086.hkfamo (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:34.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:34.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:35.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 -- 172.21.15.154:0/3393661232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 msgr2=0x7fbbcc0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:35.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3393661232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc0f9190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fbbb4009fb0 tx=0x7fbbb4009b30).stop 2023-11-15T05:05:35.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 -- 172.21.15.154:0/3393661232 shutdown_connections 2023-11-15T05:05:35.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3393661232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc0f9190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:35.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 -- 172.21.15.154:0/3393661232 >> 172.21.15.154:0/3393661232 conn(0x7fbbcc0f0b70 msgr2=0x7fbbcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:35.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 -- 172.21.15.154:0/3393661232 shutdown_connections 2023-11-15T05:05:35.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.190+0000 7fbbd231a700 1 -- 172.21.15.154:0/3393661232 wait complete. 2023-11-15T05:05:35.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbd231a700 1 Processor -- start 2023-11-15T05:05:35.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbd231a700 1 -- start start 2023-11-15T05:05:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbd231a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbd231a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc106bd0 con 0x7fbbcc0601e0 2023-11-15T05:05:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbcbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbcbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:49444/0 (socket says 172.21.15.154:49444) 2023-11-15T05:05:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbcbfff700 1 -- 172.21.15.154:0/3395839155 learned_addr learned my addr 172.21.15.154:0/3395839155 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:35.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.191+0000 7fbbcbfff700 1 -- 172.21.15.154:0/3395839155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4005040 con 0x7fbbcc0601e0 2023-11-15T05:05:35.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbcbfff700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fbbb40079e0 tx=0x7fbbb4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:35.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb4018070 con 0x7fbbcc0601e0 2023-11-15T05:05:35.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbcc106dd0 con 0x7fbbcc0601e0 2023-11-15T05:05:35.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbb4013410 con 0x7fbbcc0601e0 2023-11-15T05:05:35.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb401c3c0 con 0x7fbbcc0601e0 2023-11-15T05:05:35.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.192+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc107260 con 0x7fbbcc0601e0 2023-11-15T05:05:35.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbb401c520 con 0x7fbbcc0601e0 2023-11-15T05:05:35.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbc8ff9700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:35.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc045ad0 con 0x7fbbcc0601e0 2023-11-15T05:05:35.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbb4041680 con 0x7fbbcc0601e0 2023-11-15T05:05:35.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbcb7fe700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:35.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.193+0000 7fbbcb7fe700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 0x7fbbb8032470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbbbc0096a0 tx=0x7fbbbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:35.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.198+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbb401c920 con 0x7fbbcc0601e0 2023-11-15T05:05:35.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:34.836130+0000 mgr.smithi086.hkfamo (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:35.404 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.403+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbcc108110 con 0x7fbbcc0601e0 2023-11-15T05:05:35.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.404+0000 7fbbc8ff9700 1 -- 172.21.15.154:0/3395839155 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbbb4024300 con 0x7fbbcc0601e0 2023-11-15T05:05:35.405 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:35.405 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:35.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 msgr2=0x7fbbb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:35.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 0x7fbbb8032470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbbbc0096a0 tx=0x7fbbbc006b40).stop 2023-11-15T05:05:35.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 msgr2=0x7fbbcc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:35.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fbbb40079e0 tx=0x7fbbb4009b30).stop 2023-11-15T05:05:35.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 shutdown_connections 2023-11-15T05:05:35.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbbb802ffb0 0x7fbbb8032470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:35.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 --2- 172.21.15.154:0/3395839155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbcc0601e0 0x7fbbcc1066e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:35.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 >> 172.21.15.154:0/3395839155 conn(0x7fbbcc0f0b70 msgr2=0x7fbbcc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:35.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 shutdown_connections 2023-11-15T05:05:35.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:35.406+0000 7fbbd231a700 1 -- 172.21.15.154:0/3395839155 wait complete. 2023-11-15T05:05:35.409 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:36.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:35.405138+0000 mon.smithi086 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.154:0/3395839155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:36.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:36.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:36.836446+0000 mgr.smithi086.hkfamo (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:37.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 -- 172.21.15.154:0/988474385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 msgr2=0x7fe0040f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:37.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 --2- 172.21.15.154:0/988474385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe0040f8f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fdff4009fb0 tx=0x7fdff4009b30).stop 2023-11-15T05:05:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 -- 172.21.15.154:0/988474385 shutdown_connections 2023-11-15T05:05:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 --2- 172.21.15.154:0/988474385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe0040f8f80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 -- 172.21.15.154:0/988474385 >> 172.21.15.154:0/988474385 conn(0x7fe0040f09a0 msgr2=0x7fe0040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 -- 172.21.15.154:0/988474385 shutdown_connections 2023-11-15T05:05:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 -- 172.21.15.154:0/988474385 wait complete. 2023-11-15T05:05:37.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.980+0000 7fe008d57700 1 Processor -- start 2023-11-15T05:05:37.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe008d57700 1 -- start start 2023-11-15T05:05:37.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe008d57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:37.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe008d57700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe004106a20 con 0x7fe004060180 2023-11-15T05:05:37.984 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe0037fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:37.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe0037fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:49460/0 (socket says 172.21.15.154:49460) 2023-11-15T05:05:37.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe0037fe700 1 -- 172.21.15.154:0/3539593528 learned_addr learned my addr 172.21.15.154:0/3539593528 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:37.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe0037fe700 1 -- 172.21.15.154:0/3539593528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4005040 con 0x7fe004060180 2023-11-15T05:05:37.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.981+0000 7fe0037fe700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdff4013510 tx=0x7fdff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:37.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff4009320 con 0x7fe004060180 2023-11-15T05:05:37.986 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe004106c20 con 0x7fe004060180 2023-11-15T05:05:37.986 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdff4013f00 con 0x7fe004060180 2023-11-15T05:05:37.986 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff4014d90 con 0x7fe004060180 2023-11-15T05:05:37.986 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0041070b0 con 0x7fe004060180 2023-11-15T05:05:37.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe004045ad0 con 0x7fe004060180 2023-11-15T05:05:37.987 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.982+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdff40095f0 con 0x7fe004060180 2023-11-15T05:05:37.988 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.983+0000 7fdfebfff700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:37.988 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.983+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdff40410c0 con 0x7fe004060180 2023-11-15T05:05:37.988 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.983+0000 7fe002ffd700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:37.988 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.983+0000 7fe002ffd700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 0x7fdfec032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdff80096a0 tx=0x7fdff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:37.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:37.988+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdff4014420 con 0x7fe004060180 2023-11-15T05:05:38.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.197+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe004107f60 con 0x7fe004060180 2023-11-15T05:05:38.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.197+0000 7fdfebfff700 1 -- 172.21.15.154:0/3539593528 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdff4010230 con 0x7fe004060180 2023-11-15T05:05:38.198 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:38.198 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:38.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 msgr2=0x7fdfec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:38.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 0x7fdfec032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fdff80096a0 tx=0x7fdff8006b40).stop 2023-11-15T05:05:38.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 msgr2=0x7fe004106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:38.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdff4013510 tx=0x7fdff400b040).stop 2023-11-15T05:05:38.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 shutdown_connections 2023-11-15T05:05:38.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdfec02ff60 0x7fdfec032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:38.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 --2- 172.21.15.154:0/3539593528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe004060180 0x7fe004106530 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:38.201 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 >> 172.21.15.154:0/3539593528 conn(0x7fe0040f09a0 msgr2=0x7fe0040f1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:38.201 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 shutdown_connections 2023-11-15T05:05:38.201 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:38.199+0000 7fe008d57700 1 -- 172.21.15.154:0/3539593528 wait complete. 2023-11-15T05:05:38.201 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:38.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:38.198195+0000 mon.smithi086 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.154:0/3539593528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:39.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:39.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:39.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:38.836702+0000 mgr.smithi086.hkfamo (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:40.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 -- 172.21.15.154:0/1698850379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 msgr2=0x7fe5ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:40.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/1698850379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec0f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fe5e0009fb0 tx=0x7fe5e0009b30).stop 2023-11-15T05:05:40.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 -- 172.21.15.154:0/1698850379 shutdown_connections 2023-11-15T05:05:40.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/1698850379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec0f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:40.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 -- 172.21.15.154:0/1698850379 >> 172.21.15.154:0/1698850379 conn(0x7fe5ec0f1c70 msgr2=0x7fe5ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:40.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 -- 172.21.15.154:0/1698850379 shutdown_connections 2023-11-15T05:05:40.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.786+0000 7fe5f1c88700 1 -- 172.21.15.154:0/1698850379 wait complete. 2023-11-15T05:05:40.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.787+0000 7fe5f1c88700 1 Processor -- start 2023-11-15T05:05:40.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.787+0000 7fe5f1c88700 1 -- start start 2023-11-15T05:05:40.790 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.787+0000 7fe5f1c88700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:40.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.787+0000 7fe5f1c88700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec106c20 con 0x7fe5ec0f6510 2023-11-15T05:05:40.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.787+0000 7fe5eb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:40.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5eb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:49478/0 (socket says 172.21.15.154:49478) 2023-11-15T05:05:40.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5eb7fe700 1 -- 172.21.15.154:0/917543909 learned_addr learned my addr 172.21.15.154:0/917543909 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:40.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5eb7fe700 1 -- 172.21.15.154:0/917543909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0005040 con 0x7fe5ec0f6510 2023-11-15T05:05:40.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5eb7fe700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe5e0013510 tx=0x7fe5e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:40.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e0009320 con 0x7fe5ec0f6510 2023-11-15T05:05:40.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec106e20 con 0x7fe5ec0f6510 2023-11-15T05:05:40.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5e0013f00 con 0x7fe5ec0f6510 2023-11-15T05:05:40.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e0014d90 con 0x7fe5ec0f6510 2023-11-15T05:05:40.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.788+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec1072b0 con 0x7fe5ec0f6510 2023-11-15T05:05:40.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.789+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5e00095f0 con 0x7fe5ec0f6510 2023-11-15T05:05:40.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.789+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec045ad0 con 0x7fe5ec0f6510 2023-11-15T05:05:40.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.789+0000 7fe5d3fff700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 0x7fe5d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:40.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.789+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5e0040fe0 con 0x7fe5ec0f6510 2023-11-15T05:05:40.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.789+0000 7fe5eaffd700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 0x7fe5d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:40.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.790+0000 7fe5eaffd700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 0x7fe5d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe5dc0096a0 tx=0x7fe5dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:40.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.794+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5e001d7f0 con 0x7fe5ec0f6510 2023-11-15T05:05:41.000 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:40.999+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5ec0fbf60 con 0x7fe5ec0f6510 2023-11-15T05:05:41.001 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.000+0000 7fe5d3fff700 1 -- 172.21.15.154:0/917543909 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5e0010230 con 0x7fe5ec0f6510 2023-11-15T05:05:41.001 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:41.001 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:41.002 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 msgr2=0x7fe5d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:41.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 0x7fe5d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe5dc0096a0 tx=0x7fe5dc006b40).stop 2023-11-15T05:05:41.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 msgr2=0x7fe5ec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:41.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe5e0013510 tx=0x7fe5e000b040).stop 2023-11-15T05:05:41.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 shutdown_connections 2023-11-15T05:05:41.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe5d402ff60 0x7fe5d4032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:41.004 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 --2- 172.21.15.154:0/917543909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5ec0f6510 0x7fe5ec106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:41.004 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 >> 172.21.15.154:0/917543909 conn(0x7fe5ec0f1c70 msgr2=0x7fe5ec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:41.004 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 shutdown_connections 2023-11-15T05:05:41.004 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:41.002+0000 7fe5f1c88700 1 -- 172.21.15.154:0/917543909 wait complete. 2023-11-15T05:05:41.005 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:40.837018+0000 mgr.smithi086.hkfamo (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:42.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:41 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:41.001135+0000 mon.smithi086 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.154:0/917543909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:42.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:42.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:43.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:42.837309+0000 mgr.smithi086.hkfamo (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:43.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- 172.21.15.154:0/935039954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 msgr2=0x7f5d2c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:43.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 --2- 172.21.15.154:0/935039954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c0f90f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f5d20009fb0 tx=0x7f5d20009b30).stop 2023-11-15T05:05:43.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- 172.21.15.154:0/935039954 shutdown_connections 2023-11-15T05:05:43.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 --2- 172.21.15.154:0/935039954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c0f90f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:43.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- 172.21.15.154:0/935039954 >> 172.21.15.154:0/935039954 conn(0x7f5d2c0f0b70 msgr2=0x7f5d2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:43.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- 172.21.15.154:0/935039954 shutdown_connections 2023-11-15T05:05:43.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- 172.21.15.154:0/935039954 wait complete. 2023-11-15T05:05:43.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 Processor -- start 2023-11-15T05:05:43.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.554+0000 7f5d337ce700 1 -- start start 2023-11-15T05:05:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d337ce700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d337ce700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c106bd0 con 0x7f5d2c060280 2023-11-15T05:05:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d3156a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d3156a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:33988/0 (socket says 172.21.15.154:33988) 2023-11-15T05:05:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d3156a700 1 -- 172.21.15.154:0/3093628624 learned_addr learned my addr 172.21.15.154:0/3093628624 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:43.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d3156a700 1 -- 172.21.15.154:0/3093628624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20005040 con 0x7f5d2c060280 2023-11-15T05:05:43.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.555+0000 7f5d3156a700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5d20013510 tx=0x7f5d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:43.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.556+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d20009320 con 0x7f5d2c060280 2023-11-15T05:05:43.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.556+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c106dd0 con 0x7f5d2c060280 2023-11-15T05:05:43.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.556+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d20013f00 con 0x7f5d2c060280 2023-11-15T05:05:43.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.556+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d20014dc0 con 0x7f5d2c060280 2023-11-15T05:05:43.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.556+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c107260 con 0x7f5d2c060280 2023-11-15T05:05:43.564 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.557+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d200095f0 con 0x7f5d2c060280 2023-11-15T05:05:43.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.557+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c045ad0 con 0x7f5d2c060280 2023-11-15T05:05:43.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.557+0000 7f5d1dffb700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 0x7f5d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:43.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.557+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d20041040 con 0x7f5d2c060280 2023-11-15T05:05:43.565 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.557+0000 7f5d30d69700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 0x7f5d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:43.566 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.558+0000 7f5d30d69700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 0x7f5d18032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5d280096a0 tx=0x7f5d28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:43.566 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.562+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d20014420 con 0x7f5d2c060280 2023-11-15T05:05:43.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.765+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d2c108110 con 0x7f5d2c060280 2023-11-15T05:05:43.767 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.766+0000 7f5d1dffb700 1 -- 172.21.15.154:0/3093628624 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d20010230 con 0x7f5d2c060280 2023-11-15T05:05:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:43.768 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:43.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 msgr2=0x7f5d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:43.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 0x7f5d18032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5d280096a0 tx=0x7f5d28006b40).stop 2023-11-15T05:05:43.769 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 msgr2=0x7f5d2c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:43.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f5d20013510 tx=0x7f5d2000b040).stop 2023-11-15T05:05:43.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 shutdown_connections 2023-11-15T05:05:43.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d1802ff60 0x7f5d18032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:43.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 --2- 172.21.15.154:0/3093628624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d2c060280 0x7f5d2c1066e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:43.770 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 >> 172.21.15.154:0/3093628624 conn(0x7f5d2c0f0b70 msgr2=0x7f5d2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:43.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 shutdown_connections 2023-11-15T05:05:43.771 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:43.767+0000 7f5d337ce700 1 -- 172.21.15.154:0/3093628624 wait complete. 2023-11-15T05:05:43.771 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:44.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:43.766864+0000 mon.smithi086 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.154:0/3093628624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:45.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:45.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:44.837586+0000 mgr.smithi086.hkfamo (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:46.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- 172.21.15.154:0/246186969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 msgr2=0x7f14940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:46.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 --2- 172.21.15.154:0/246186969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f14940f6930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f1484009fb0 tx=0x7f1484009b30).stop 2023-11-15T05:05:46.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- 172.21.15.154:0/246186969 shutdown_connections 2023-11-15T05:05:46.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 --2- 172.21.15.154:0/246186969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f14940f6930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:46.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- 172.21.15.154:0/246186969 >> 172.21.15.154:0/246186969 conn(0x7f14940f1c70 msgr2=0x7f14940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:46.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- 172.21.15.154:0/246186969 shutdown_connections 2023-11-15T05:05:46.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- 172.21.15.154:0/246186969 wait complete. 2023-11-15T05:05:46.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 Processor -- start 2023-11-15T05:05:46.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- start start 2023-11-15T05:05:46.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:46.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.294+0000 7f149a120700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1494069e20 con 0x7f14940f6510 2023-11-15T05:05:46.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f14937fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:46.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f14937fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34014/0 (socket says 172.21.15.154:34014) 2023-11-15T05:05:46.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f14937fe700 1 -- 172.21.15.154:0/1937597232 learned_addr learned my addr 172.21.15.154:0/1937597232 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:46.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f14937fe700 1 -- 172.21.15.154:0/1937597232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484005040 con 0x7f14940f6510 2023-11-15T05:05:46.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f14937fe700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1484013510 tx=0x7f148400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:46.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484009320 con 0x7f14940f6510 2023-11-15T05:05:46.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1494069fc0 con 0x7f14940f6510 2023-11-15T05:05:46.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.295+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1484013f00 con 0x7f14940f6510 2023-11-15T05:05:46.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.296+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484014d90 con 0x7f14940f6510 2023-11-15T05:05:46.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.296+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149406a450 con 0x7f14940f6510 2023-11-15T05:05:46.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.296+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14840263a0 con 0x7f14940f6510 2023-11-15T05:05:46.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.296+0000 7f147bfff700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 0x7f147c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:46.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.297+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1484041100 con 0x7f14940f6510 2023-11-15T05:05:46.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.297+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1494059100 con 0x7f14940f6510 2023-11-15T05:05:46.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.297+0000 7f1492ffd700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 0x7f147c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:46.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.297+0000 7f1492ffd700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 0x7f147c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f14880096a0 tx=0x7f1488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:46.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.301+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1484012920 con 0x7f14940f6510 2023-11-15T05:05:46.508 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.508+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f149406ab50 con 0x7f14940f6510 2023-11-15T05:05:46.509 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.508+0000 7f147bfff700 1 -- 172.21.15.154:0/1937597232 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f148402b020 con 0x7f14940f6510 2023-11-15T05:05:46.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:46.510 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:46.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 msgr2=0x7f147c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:46.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 0x7f147c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f14880096a0 tx=0x7f1488006b40).stop 2023-11-15T05:05:46.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 msgr2=0x7f149406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:46.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f1484013510 tx=0x7f148400b040).stop 2023-11-15T05:05:46.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 shutdown_connections 2023-11-15T05:05:46.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f147c02ff60 0x7f147c032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:46.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 --2- 172.21.15.154:0/1937597232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14940f6510 0x7f149406b9a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:46.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 >> 172.21.15.154:0/1937597232 conn(0x7f14940f1c70 msgr2=0x7f14940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:46.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 shutdown_connections 2023-11-15T05:05:46.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:46.510+0000 7f149a120700 1 -- 172.21.15.154:0/1937597232 wait complete. 2023-11-15T05:05:46.514 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:47.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:46.509420+0000 mon.smithi086 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.154:0/1937597232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:47.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:46.837923+0000 mgr.smithi086.hkfamo (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:47.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:47.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:49.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 -- 172.21.15.154:0/4201515003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 msgr2=0x7f02080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:49.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 --2- 172.21.15.154:0/4201515003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f02080f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f01fc009fb0 tx=0x7f01fc009b30).stop 2023-11-15T05:05:49.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 -- 172.21.15.154:0/4201515003 shutdown_connections 2023-11-15T05:05:49.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 --2- 172.21.15.154:0/4201515003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f02080f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:49.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 -- 172.21.15.154:0/4201515003 >> 172.21.15.154:0/4201515003 conn(0x7f02080f0b70 msgr2=0x7f02080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:49.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 -- 172.21.15.154:0/4201515003 shutdown_connections 2023-11-15T05:05:49.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.105+0000 7f020d46e700 1 -- 172.21.15.154:0/4201515003 wait complete. 2023-11-15T05:05:49.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.106+0000 7f020d46e700 1 Processor -- start 2023-11-15T05:05:49.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.106+0000 7f020d46e700 1 -- start start 2023-11-15T05:05:49.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.106+0000 7f020d46e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:49.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.106+0000 7f020d46e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0208060240 con 0x7f02080f5410 2023-11-15T05:05:49.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.106+0000 7f0206ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:49.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f0206ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34036/0 (socket says 172.21.15.154:34036) 2023-11-15T05:05:49.112 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f0206ffd700 1 -- 172.21.15.154:0/1130993110 learned_addr learned my addr 172.21.15.154:0/1130993110 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:49.113 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f0206ffd700 1 -- 172.21.15.154:0/1130993110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc005040 con 0x7f02080f5410 2023-11-15T05:05:49.113 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f0206ffd700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f01fc013510 tx=0x7f01fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:49.113 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01fc009320 con 0x7f02080f5410 2023-11-15T05:05:49.114 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0208060440 con 0x7f02080f5410 2023-11-15T05:05:49.114 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01fc013f00 con 0x7f02080f5410 2023-11-15T05:05:49.114 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01fc014d90 con 0x7f02080f5410 2023-11-15T05:05:49.115 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.107+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0208109ab0 con 0x7f02080f5410 2023-11-15T05:05:49.116 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.108+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01fc0263a0 con 0x7f02080f5410 2023-11-15T05:05:49.117 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.108+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0208045ad0 con 0x7f02080f5410 2023-11-15T05:05:49.117 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.108+0000 7f01ef7fe700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 0x7f01f00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:49.117 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.108+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01fc041060 con 0x7f02080f5410 2023-11-15T05:05:49.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.109+0000 7f02067fc700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 0x7f01f00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:49.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.109+0000 7f02067fc700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 0x7f01f00324c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f01f80096a0 tx=0x7f01f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:49.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.113+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01fc06d2b0 con 0x7f02080f5410 2023-11-15T05:05:49.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:48.838304+0000 mgr.smithi086.hkfamo (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:49.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.317+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02080fae60 con 0x7f02080f5410 2023-11-15T05:05:49.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.318+0000 7f01ef7fe700 1 -- 172.21.15.154:0/1130993110 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01fc0126f0 con 0x7f02080f5410 2023-11-15T05:05:49.319 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:49.319 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 msgr2=0x7f01f00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 0x7f01f00324c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f01f80096a0 tx=0x7f01f8006b40).stop 2023-11-15T05:05:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 msgr2=0x7f0208109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f01fc013510 tx=0x7f01fc00b040).stop 2023-11-15T05:05:49.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 shutdown_connections 2023-11-15T05:05:49.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f01f0030000 0x7f01f00324c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:49.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 --2- 172.21.15.154:0/1130993110 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02080f5410 0x7f0208109400 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:49.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 >> 172.21.15.154:0/1130993110 conn(0x7f02080f0b70 msgr2=0x7f02080f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:49.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 shutdown_connections 2023-11-15T05:05:49.323 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:49.320+0000 7f020d46e700 1 -- 172.21.15.154:0/1130993110 wait complete. 2023-11-15T05:05:49.323 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:50.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:49.319109+0000 mon.smithi086 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.154:0/1130993110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:50.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:50.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:51.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:50.838650+0000 mgr.smithi086.hkfamo (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:51.940 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 -- 172.21.15.154:0/513656807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0f5410 msgr2=0x7f5e4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:51.940 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 --2- 172.21.15.154:0/513656807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0f5410 0x7f5e4c0f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f5e48009fb0 tx=0x7f5e48009b30).stop 2023-11-15T05:05:51.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 -- 172.21.15.154:0/513656807 shutdown_connections 2023-11-15T05:05:51.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 --2- 172.21.15.154:0/513656807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c0f5410 0x7f5e4c0f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:51.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 -- 172.21.15.154:0/513656807 >> 172.21.15.154:0/513656807 conn(0x7f5e4c0f0b70 msgr2=0x7f5e4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:51.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 -- 172.21.15.154:0/513656807 shutdown_connections 2023-11-15T05:05:51.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.938+0000 7f5e535d6700 1 -- 172.21.15.154:0/513656807 wait complete. 2023-11-15T05:05:51.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e535d6700 1 Processor -- start 2023-11-15T05:05:51.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e535d6700 1 -- start start 2023-11-15T05:05:51.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e535d6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:51.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e535d6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c107110 con 0x7f5e4c106800 2023-11-15T05:05:51.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e51372700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:51.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e51372700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34062/0 (socket says 172.21.15.154:34062) 2023-11-15T05:05:51.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.939+0000 7f5e51372700 1 -- 172.21.15.154:0/704455838 learned_addr learned my addr 172.21.15.154:0/704455838 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:51.945 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e51372700 1 -- 172.21.15.154:0/704455838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48005040 con 0x7f5e4c106800 2023-11-15T05:05:51.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e51372700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5e48009f80 tx=0x7f5e48009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:51.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e480136c0 con 0x7f5e4c106800 2023-11-15T05:05:51.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e48013e70 con 0x7f5e4c106800 2023-11-15T05:05:51.947 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e48014420 con 0x7f5e4c106800 2023-11-15T05:05:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.940+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c107310 con 0x7f5e4c106800 2023-11-15T05:05:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.941+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c107e60 con 0x7f5e4c106800 2023-11-15T05:05:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.941+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c059100 con 0x7f5e4c106800 2023-11-15T05:05:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.942+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e48025060 con 0x7f5e4c106800 2023-11-15T05:05:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.942+0000 7f5e41ffb700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 0x7f5e38032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:51.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.942+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e48040d80 con 0x7f5e4c106800 2023-11-15T05:05:51.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.942+0000 7f5e50b71700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 0x7f5e38032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:51.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.942+0000 7f5e50b71700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 0x7f5e38032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5e3c0096a0 tx=0x7f5e3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:51.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:51.946+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e48044030 con 0x7f5e4c106800 2023-11-15T05:05:52.152 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.149+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e4c045ad0 con 0x7f5e4c106800 2023-11-15T05:05:52.153 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.150+0000 7f5e41ffb700 1 -- 172.21.15.154:0/704455838 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e4802d2d0 con 0x7f5e4c106800 2023-11-15T05:05:52.153 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:52.154 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:52.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 msgr2=0x7f5e38032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:52.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 0x7f5e38032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5e3c0096a0 tx=0x7f5e3c006b40).stop 2023-11-15T05:05:52.155 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 msgr2=0x7f5e4c106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:52.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5e48009f80 tx=0x7f5e48009b30).stop 2023-11-15T05:05:52.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 shutdown_connections 2023-11-15T05:05:52.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5e3802ffb0 0x7f5e38032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:52.156 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 --2- 172.21.15.154:0/704455838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e4c106800 0x7f5e4c106c20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:52.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 >> 172.21.15.154:0/704455838 conn(0x7f5e4c0f0b70 msgr2=0x7f5e4c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:52.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 shutdown_connections 2023-11-15T05:05:52.158 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:52.152+0000 7f5e535d6700 1 -- 172.21.15.154:0/704455838 wait complete. 2023-11-15T05:05:52.158 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:52.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:52.151252+0000 mon.smithi086 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.154:0/704455838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:53.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:53.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:53.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:52.838944+0000 mgr.smithi086.hkfamo (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:54.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 -- 172.21.15.154:0/1751287444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 msgr2=0x7f1b580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:54.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 --2- 172.21.15.154:0/1751287444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f90f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1b48009fb0 tx=0x7f1b48009b30).stop 2023-11-15T05:05:54.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 -- 172.21.15.154:0/1751287444 shutdown_connections 2023-11-15T05:05:54.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 --2- 172.21.15.154:0/1751287444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f90f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:54.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 -- 172.21.15.154:0/1751287444 >> 172.21.15.154:0/1751287444 conn(0x7f1b580f0b70 msgr2=0x7f1b580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:54.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 -- 172.21.15.154:0/1751287444 shutdown_connections 2023-11-15T05:05:54.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.722+0000 7f1b5f576700 1 -- 172.21.15.154:0/1751287444 wait complete. 2023-11-15T05:05:54.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5f576700 1 Processor -- start 2023-11-15T05:05:54.728 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5f576700 1 -- start start 2023-11-15T05:05:54.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5f576700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:54.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5f576700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580f4ff0 con 0x7f1b58060280 2023-11-15T05:05:54.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5d312700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:54.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5d312700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54228/0 (socket says 172.21.15.154:54228) 2023-11-15T05:05:54.729 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5d312700 1 -- 172.21.15.154:0/2341828686 learned_addr learned my addr 172.21.15.154:0/2341828686 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:54.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5d312700 1 -- 172.21.15.154:0/2341828686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b48005040 con 0x7f1b58060280 2023-11-15T05:05:54.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.723+0000 7f1b5d312700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1b48013510 tx=0x7f1b4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:54.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.724+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b48009320 con 0x7f1b58060280 2023-11-15T05:05:54.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.724+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b48013f00 con 0x7f1b58060280 2023-11-15T05:05:54.730 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.724+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b580f5190 con 0x7f1b58060280 2023-11-15T05:05:54.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.724+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b48014dc0 con 0x7f1b58060280 2023-11-15T05:05:54.731 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.724+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b580f5620 con 0x7f1b58060280 2023-11-15T05:05:54.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.725+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b58059100 con 0x7f1b58060280 2023-11-15T05:05:54.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.725+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b480095f0 con 0x7f1b58060280 2023-11-15T05:05:54.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.725+0000 7f1b4dffb700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 0x7f1b44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:54.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.725+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b480410e0 con 0x7f1b58060280 2023-11-15T05:05:54.732 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.725+0000 7f1b5cb11700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 0x7f1b44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:54.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.726+0000 7f1b5cb11700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 0x7f1b44032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1b540096a0 tx=0x7f1b54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:54.733 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.730+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b48014420 con 0x7f1b58060280 2023-11-15T05:05:54.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.937+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b580fbf50 con 0x7f1b58060280 2023-11-15T05:05:54.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.937+0000 7f1b4dffb700 1 -- 172.21.15.154:0/2341828686 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b4801c920 con 0x7f1b58060280 2023-11-15T05:05:54.939 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:54.939 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:54.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 msgr2=0x7f1b44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:54.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 0x7f1b44032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1b540096a0 tx=0x7f1b54006b40).stop 2023-11-15T05:05:54.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 msgr2=0x7f1b580f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:54.943 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1b48013510 tx=0x7f1b4800b040).stop 2023-11-15T05:05:54.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 shutdown_connections 2023-11-15T05:05:54.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b4402ff60 0x7f1b44032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:54.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 --2- 172.21.15.154:0/2341828686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58060280 0x7f1b580f6b70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:54.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 >> 172.21.15.154:0/2341828686 conn(0x7f1b580f0b70 msgr2=0x7f1b580f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:54.944 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 shutdown_connections 2023-11-15T05:05:54.945 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:54.939+0000 7f1b5f576700 1 -- 172.21.15.154:0/2341828686 wait complete. 2023-11-15T05:05:54.945 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:54.839249+0000 mgr.smithi086.hkfamo (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:56.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:55 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:54.938570+0000 mon.smithi086 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.154:0/2341828686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:56.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:56.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:57.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.525+0000 7f5dacebd700 1 -- 172.21.15.154:0/2685004544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 msgr2=0x7f5da80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:57.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.525+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2685004544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da80f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f5d98009fb0 tx=0x7f5d98009b30).stop 2023-11-15T05:05:57.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 -- 172.21.15.154:0/2685004544 shutdown_connections 2023-11-15T05:05:57.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2685004544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da80f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:57.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 -- 172.21.15.154:0/2685004544 >> 172.21.15.154:0/2685004544 conn(0x7f5da80f0b70 msgr2=0x7f5da80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:57.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 -- 172.21.15.154:0/2685004544 shutdown_connections 2023-11-15T05:05:57.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 -- 172.21.15.154:0/2685004544 wait complete. 2023-11-15T05:05:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 Processor -- start 2023-11-15T05:05:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 -- start start 2023-11-15T05:05:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.526+0000 7f5dacebd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5dacebd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8105b20 con 0x7f5da80f5410 2023-11-15T05:05:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5da659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:57.531 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5da659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54252/0 (socket says 172.21.15.154:54252) 2023-11-15T05:05:57.531 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5da659c700 1 -- 172.21.15.154:0/2165082380 learned_addr learned my addr 172.21.15.154:0/2165082380 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:05:57.531 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5da659c700 1 -- 172.21.15.154:0/2165082380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98005040 con 0x7f5da80f5410 2023-11-15T05:05:57.531 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5da659c700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f5d98013510 tx=0x7f5d9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:57.531 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.527+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d98009320 con 0x7f5da80f5410 2023-11-15T05:05:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da8105d20 con 0x7f5da80f5410 2023-11-15T05:05:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d98013f00 con 0x7f5da80f5410 2023-11-15T05:05:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d98014d90 con 0x7f5da80f5410 2023-11-15T05:05:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da81061b0 con 0x7f5da80f5410 2023-11-15T05:05:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da8045ad0 con 0x7f5da80f5410 2023-11-15T05:05:57.533 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d980095f0 con 0x7f5da80f5410 2023-11-15T05:05:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.528+0000 7f5d9effd700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 0x7f5d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:05:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.529+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d980410a0 con 0x7f5da80f5410 2023-11-15T05:05:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.529+0000 7f5da5d9b700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 0x7f5d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:05:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.529+0000 7f5da5d9b700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 0x7f5d94032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f5d900096a0 tx=0x7f5d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:05:57.535 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.533+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d98014420 con 0x7f5da80f5410 2023-11-15T05:05:57.738 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.737+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5da80fae60 con 0x7f5da80f5410 2023-11-15T05:05:57.738 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.738+0000 7f5d9effd700 1 -- 172.21.15.154:0/2165082380 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d98010230 con 0x7f5da80f5410 2023-11-15T05:05:57.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:05:57.739 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:05:57.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:56.839562+0000 mgr.smithi086.hkfamo (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:05:57.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 msgr2=0x7f5d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:57.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 0x7f5d94032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f5d900096a0 tx=0x7f5d90006b40).stop 2023-11-15T05:05:57.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 msgr2=0x7f5da8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:05:57.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f5d98013510 tx=0x7f5d9800b040).stop 2023-11-15T05:05:57.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 shutdown_connections 2023-11-15T05:05:57.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5d9402ff60 0x7f5d94032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:57.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 --2- 172.21.15.154:0/2165082380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5da80f5410 0x7f5da8105630 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:05:57.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.739+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 >> 172.21.15.154:0/2165082380 conn(0x7f5da80f0b70 msgr2=0x7f5da80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:05:57.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.740+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 shutdown_connections 2023-11-15T05:05:57.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:05:57.740+0000 7f5dacebd700 1 -- 172.21.15.154:0/2165082380 wait complete. 2023-11-15T05:05:57.743 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:05:58.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:05:57.739029+0000 mon.smithi086 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.154:0/2165082380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:05:59.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:05:59.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:05:59.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:05:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:05:58.839922+0000 mgr.smithi086.hkfamo (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:00.341 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.339+0000 7f5a72874700 1 -- 172.21.15.154:0/3851613651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 msgr2=0x7f5a6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:00.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.339+0000 7f5a72874700 1 --2- 172.21.15.154:0/3851613651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5a5c009fb0 tx=0x7f5a5c009b30).stop 2023-11-15T05:06:00.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- 172.21.15.154:0/3851613651 shutdown_connections 2023-11-15T05:06:00.342 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 --2- 172.21.15.154:0/3851613651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- 172.21.15.154:0/3851613651 >> 172.21.15.154:0/3851613651 conn(0x7f5a6c0f09a0 msgr2=0x7f5a6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- 172.21.15.154:0/3851613651 shutdown_connections 2023-11-15T05:06:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- 172.21.15.154:0/3851613651 wait complete. 2023-11-15T05:06:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 Processor -- start 2023-11-15T05:06:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- start start 2023-11-15T05:06:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.340+0000 7f5a72874700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0fb5a0 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a71872700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a71872700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54268/0 (socket says 172.21.15.154:54268) 2023-11-15T05:06:00.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a71872700 1 -- 172.21.15.154:0/3007687069 learned_addr learned my addr 172.21.15.154:0/3007687069 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:00.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a71872700 1 -- 172.21.15.154:0/3007687069 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c005040 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a71872700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5a5c013510 tx=0x7f5a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a5c009320 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c0fb740 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a5c013f00 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a5c014dc0 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.341+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c0fbbd0 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.342+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a5c01d3d0 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.342+0000 7f5a627fc700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 0x7f5a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:00.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.342+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a5c041030 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.342+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c059100 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.343+0000 7f5a71071700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 0x7f5a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:00.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.343+0000 7f5a71071700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 0x7f5a58032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5a680096a0 tx=0x7f5a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:00.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.347+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a5c012920 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.553 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.553+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a6c0fc2d0 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.554 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.553+0000 7f5a627fc700 1 -- 172.21.15.154:0/3007687069 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a5c012330 con 0x7f5a6c0f51a0 2023-11-15T05:06:00.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:00.555 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:00.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 msgr2=0x7f5a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:00.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 0x7f5a58032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5a680096a0 tx=0x7f5a68006b40).stop 2023-11-15T05:06:00.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 msgr2=0x7f5a6c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:00.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5a5c013510 tx=0x7f5a5c00b040).stop 2023-11-15T05:06:00.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 shutdown_connections 2023-11-15T05:06:00.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5a5802ff60 0x7f5a58032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:00.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 --2- 172.21.15.154:0/3007687069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a6c0f51a0 0x7f5a6c0fd120 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:00.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 >> 172.21.15.154:0/3007687069 conn(0x7f5a6c0f09a0 msgr2=0x7f5a6c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:00.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 shutdown_connections 2023-11-15T05:06:00.558 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:00.555+0000 7f5a72874700 1 -- 172.21.15.154:0/3007687069 wait complete. 2023-11-15T05:06:00.558 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:00.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:00.554339+0000 mon.smithi086 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.154:0/3007687069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:01.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:01.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:01.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:01 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:00.840272+0000 mgr.smithi086.hkfamo (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:03.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 -- 172.21.15.154:0/3070029722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 msgr2=0x7fce8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:03.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 --2- 172.21.15.154:0/3070029722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fce80009fb0 tx=0x7fce80009b30).stop 2023-11-15T05:06:03.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 -- 172.21.15.154:0/3070029722 shutdown_connections 2023-11-15T05:06:03.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 --2- 172.21.15.154:0/3070029722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:03.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 -- 172.21.15.154:0/3070029722 >> 172.21.15.154:0/3070029722 conn(0x7fce8c0f0b70 msgr2=0x7fce8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:03.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 -- 172.21.15.154:0/3070029722 shutdown_connections 2023-11-15T05:06:03.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.083+0000 7fce91408700 1 -- 172.21.15.154:0/3070029722 wait complete. 2023-11-15T05:06:03.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce91408700 1 Processor -- start 2023-11-15T05:06:03.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce91408700 1 -- start start 2023-11-15T05:06:03.087 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce91408700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:03.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce91408700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c060280 con 0x7fce8c0f5410 2023-11-15T05:06:03.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce8affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:03.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce8affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54296/0 (socket says 172.21.15.154:54296) 2023-11-15T05:06:03.088 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.084+0000 7fce8affd700 1 -- 172.21.15.154:0/2042044432 learned_addr learned my addr 172.21.15.154:0/2042044432 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce8affd700 1 -- 172.21.15.154:0/2042044432 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80005040 con 0x7fce8c0f5410 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce8affd700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fce80014000 tx=0x7fce8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce800095f0 con 0x7fce8c0f5410 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c060480 con 0x7fce8c0f5410 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce80007e70 con 0x7fce8c0f5410 2023-11-15T05:06:03.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c109af0 con 0x7fce8c0f5410 2023-11-15T05:06:03.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.085+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce80014e50 con 0x7fce8c0f5410 2023-11-15T05:06:03.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.086+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce80025070 con 0x7fce8c0f5410 2023-11-15T05:06:03.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.086+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c045ad0 con 0x7fce8c0f5410 2023-11-15T05:06:03.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.086+0000 7fce737fe700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 0x7fce740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:03.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.086+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce80040d20 con 0x7fce8c0f5410 2023-11-15T05:06:03.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.087+0000 7fce8a7fc700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 0x7fce740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:03.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.087+0000 7fce8a7fc700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 0x7fce740323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fce7c0096a0 tx=0x7fce7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:03.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.091+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce800254e0 con 0x7fce8c0f5410 2023-11-15T05:06:03.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:02.840649+0000 mgr.smithi086.hkfamo (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:03.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.298+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce8c10a290 con 0x7fce8c0f5410 2023-11-15T05:06:03.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.299+0000 7fce737fe700 1 -- 172.21.15.154:0/2042044432 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce80025300 con 0x7fce8c0f5410 2023-11-15T05:06:03.300 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:03.300 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:03.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 msgr2=0x7fce740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 0x7fce740323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fce7c0096a0 tx=0x7fce7c006b40).stop 2023-11-15T05:06:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 msgr2=0x7fce8c1093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fce80014000 tx=0x7fce8000b040).stop 2023-11-15T05:06:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 shutdown_connections 2023-11-15T05:06:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce7402ff10 0x7fce740323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 --2- 172.21.15.154:0/2042044432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce8c0f5410 0x7fce8c1093c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 >> 172.21.15.154:0/2042044432 conn(0x7fce8c0f0b70 msgr2=0x7fce8c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 shutdown_connections 2023-11-15T05:06:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:03.300+0000 7fce91408700 1 -- 172.21.15.154:0/2042044432 wait complete. 2023-11-15T05:06:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:04.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:03.299810+0000 mon.smithi086 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.154:0/2042044432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:04.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:04.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:04.840907+0000 mgr.smithi086.hkfamo (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:05.849 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- 172.21.15.154:0/1648070707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 msgr2=0x7f1eb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:05.849 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 --2- 172.21.15.154:0/1648070707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb80f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f1ea8009fb0 tx=0x7f1ea8009b30).stop 2023-11-15T05:06:05.849 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- 172.21.15.154:0/1648070707 shutdown_connections 2023-11-15T05:06:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 --2- 172.21.15.154:0/1648070707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb80f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- 172.21.15.154:0/1648070707 >> 172.21.15.154:0/1648070707 conn(0x7f1eb80f0b70 msgr2=0x7f1eb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- 172.21.15.154:0/1648070707 shutdown_connections 2023-11-15T05:06:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- 172.21.15.154:0/1648070707 wait complete. 2023-11-15T05:06:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 Processor -- start 2023-11-15T05:06:05.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.848+0000 7f1ebe324700 1 -- start start 2023-11-15T05:06:05.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1ebe324700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:05.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1ebe324700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb8105b20 con 0x7f1eb80f5410 2023-11-15T05:06:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58244/0 (socket says 172.21.15.154:58244) 2023-11-15T05:06:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb7fff700 1 -- 172.21.15.154:0/3784904079 learned_addr learned my addr 172.21.15.154:0/3784904079 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb7fff700 1 -- 172.21.15.154:0/3784904079 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1eb80f5410 2023-11-15T05:06:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb7fff700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1ea8013510 tx=0x7f1ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:05.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8009320 con 0x7f1eb80f5410 2023-11-15T05:06:05.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8105d20 con 0x7f1eb80f5410 2023-11-15T05:06:05.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ea8013f00 con 0x7f1eb80f5410 2023-11-15T05:06:05.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8014d90 con 0x7f1eb80f5410 2023-11-15T05:06:05.854 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.849+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb81061b0 con 0x7f1eb80f5410 2023-11-15T05:06:05.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.850+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ea80263a0 con 0x7f1eb80f5410 2023-11-15T05:06:05.855 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.850+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8045ad0 con 0x7f1eb80f5410 2023-11-15T05:06:05.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.850+0000 7f1eb4ff9700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 0x7f1ea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:05.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.850+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ea8041300 con 0x7f1eb80f5410 2023-11-15T05:06:05.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.851+0000 7f1eb77fe700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 0x7f1ea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:05.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.851+0000 7f1eb77fe700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 0x7f1ea4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ea00096a0 tx=0x7f1ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:05.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:05.855+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ea801ccf0 con 0x7f1eb80f5410 2023-11-15T05:06:06.062 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.062+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1eb80fae60 con 0x7f1eb80f5410 2023-11-15T05:06:06.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.062+0000 7f1eb4ff9700 1 -- 172.21.15.154:0/3784904079 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ea801cb10 con 0x7f1eb80f5410 2023-11-15T05:06:06.064 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:06.064 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:06.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 msgr2=0x7f1ea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:06.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 0x7f1ea4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ea00096a0 tx=0x7f1ea0006b40).stop 2023-11-15T05:06:06.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 msgr2=0x7f1eb8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:06.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1ea8013510 tx=0x7f1ea800b040).stop 2023-11-15T05:06:06.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 shutdown_connections 2023-11-15T05:06:06.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea402ff60 0x7f1ea4032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:06.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 --2- 172.21.15.154:0/3784904079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eb80f5410 0x7f1eb8105630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:06.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 >> 172.21.15.154:0/3784904079 conn(0x7f1eb80f0b70 msgr2=0x7f1eb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:06.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 shutdown_connections 2023-11-15T05:06:06.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:06.064+0000 7f1ebe324700 1 -- 172.21.15.154:0/3784904079 wait complete. 2023-11-15T05:06:06.067 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:06.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:06.063550+0000 mon.smithi086 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.154:0/3784904079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:07.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:07.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:06.841239+0000 mgr.smithi086.hkfamo (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- 172.21.15.154:0/2748672169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 msgr2=0x7fa0cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/2748672169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc0f66c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fa0bc009fb0 tx=0x7fa0bc009b30).stop 2023-11-15T05:06:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- 172.21.15.154:0/2748672169 shutdown_connections 2023-11-15T05:06:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/2748672169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc0f66c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- 172.21.15.154:0/2748672169 >> 172.21.15.154:0/2748672169 conn(0x7fa0cc0f1ac0 msgr2=0x7fa0cc0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- 172.21.15.154:0/2748672169 shutdown_connections 2023-11-15T05:06:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- 172.21.15.154:0/2748672169 wait complete. 2023-11-15T05:06:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 Processor -- start 2023-11-15T05:06:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.657+0000 7fa0cbfff700 1 -- start start 2023-11-15T05:06:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0cbfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0cbfff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0cc106990 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0caffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0caffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58250/0 (socket says 172.21.15.154:58250) 2023-11-15T05:06:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0caffd700 1 -- 172.21.15.154:0/1238085673 learned_addr learned my addr 172.21.15.154:0/1238085673 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:08.662 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0caffd700 1 -- 172.21.15.154:0/1238085673 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc005040 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0caffd700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa0bc013510 tx=0x7fa0bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0bc009320 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc106b90 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0bc013f00 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.658+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0bc014dc0 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.659+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc107020 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.659+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0bc014420 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.659+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc045ad0 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.659+0000 7fa0b37fe700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 0x7fa0b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.659+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0bc041050 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.660+0000 7fa0ca7fc700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 0x7fa0b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.660+0000 7fa0ca7fc700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 0x7fa0b4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fa0c00096a0 tx=0x7fa0c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.664+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0bc0095f0 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.869+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0cc107ed0 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.869+0000 7fa0b37fe700 1 -- 172.21.15.154:0/1238085673 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0bc01cb10 con 0x7fa0cc0f62a0 2023-11-15T05:06:08.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:08.872 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:08.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 msgr2=0x7fa0b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:08.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 0x7fa0b4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fa0c00096a0 tx=0x7fa0c0006b40).stop 2023-11-15T05:06:08.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 msgr2=0x7fa0cc1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:08.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa0bc013510 tx=0x7fa0bc00b040).stop 2023-11-15T05:06:08.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 shutdown_connections 2023-11-15T05:06:08.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa0b402ff60 0x7fa0b4032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:08.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 --2- 172.21.15.154:0/1238085673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0cc0f62a0 0x7fa0cc1064a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:08.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 >> 172.21.15.154:0/1238085673 conn(0x7fa0cc0f1ac0 msgr2=0x7fa0cc0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:08.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 shutdown_connections 2023-11-15T05:06:08.875 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:08.871+0000 7fa0cbfff700 1 -- 172.21.15.154:0/1238085673 wait complete. 2023-11-15T05:06:08.875 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:09.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:08.841532+0000 mgr.smithi086.hkfamo (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:09.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:08.871527+0000 mon.smithi086 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.154:0/1238085673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:10.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:10.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:10.841828+0000 mgr.smithi086.hkfamo (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:11.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 -- 172.21.15.154:0/152687728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 msgr2=0x7f91340f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:11.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 --2- 172.21.15.154:0/152687728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f8f20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f9124009fb0 tx=0x7f9124009b30).stop 2023-11-15T05:06:11.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 -- 172.21.15.154:0/152687728 shutdown_connections 2023-11-15T05:06:11.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 --2- 172.21.15.154:0/152687728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f8f20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:11.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 -- 172.21.15.154:0/152687728 >> 172.21.15.154:0/152687728 conn(0x7f91340f09a0 msgr2=0x7f91340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:11.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 -- 172.21.15.154:0/152687728 shutdown_connections 2023-11-15T05:06:11.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.400+0000 7f9133fff700 1 -- 172.21.15.154:0/152687728 wait complete. 2023-11-15T05:06:11.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9133fff700 1 Processor -- start 2023-11-15T05:06:11.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9133fff700 1 -- start start 2023-11-15T05:06:11.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9133fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:11.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9133fff700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340f4e20 con 0x7f91340601e0 2023-11-15T05:06:11.408 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9132ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:11.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9132ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:58270/0 (socket says 172.21.15.154:58270) 2023-11-15T05:06:11.409 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9132ffd700 1 -- 172.21.15.154:0/230482188 learned_addr learned my addr 172.21.15.154:0/230482188 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:11.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9132ffd700 1 -- 172.21.15.154:0/230482188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124005040 con 0x7f91340601e0 2023-11-15T05:06:11.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.401+0000 7f9132ffd700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f9124014000 tx=0x7f912400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:11.410 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91240095f0 con 0x7f91340601e0 2023-11-15T05:06:11.411 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9124007e70 con 0x7f91340601e0 2023-11-15T05:06:11.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91340f4fc0 con 0x7f91340601e0 2023-11-15T05:06:11.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9124014ea0 con 0x7f91340601e0 2023-11-15T05:06:11.412 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91340f5450 con 0x7f91340601e0 2023-11-15T05:06:11.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9124025070 con 0x7f91340601e0 2023-11-15T05:06:11.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.402+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9134059100 con 0x7f91340601e0 2023-11-15T05:06:11.413 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.403+0000 7f911b7fe700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 0x7f911c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:11.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.403+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91240409f0 con 0x7f91340601e0 2023-11-15T05:06:11.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.403+0000 7f91327fc700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 0x7f911c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:11.414 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.403+0000 7f91327fc700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 0x7f911c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f91280096a0 tx=0x7f9128006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:11.415 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.408+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91240079c0 con 0x7f91340601e0 2023-11-15T05:06:11.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.614+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91340fbcd0 con 0x7f91340601e0 2023-11-15T05:06:11.616 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.615+0000 7f911b7fe700 1 -- 172.21.15.154:0/230482188 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f912400f300 con 0x7f91340601e0 2023-11-15T05:06:11.617 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:11.617 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:11.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 msgr2=0x7f911c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:11.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 0x7f911c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f91280096a0 tx=0x7f9128006b40).stop 2023-11-15T05:06:11.618 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 msgr2=0x7f91340f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f9124014000 tx=0x7f912400b040).stop 2023-11-15T05:06:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 shutdown_connections 2023-11-15T05:06:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f911c02ff60 0x7f911c032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 --2- 172.21.15.154:0/230482188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91340601e0 0x7f91340f69a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 >> 172.21.15.154:0/230482188 conn(0x7f91340f09a0 msgr2=0x7f91340f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:11.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 shutdown_connections 2023-11-15T05:06:11.620 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:11.616+0000 7f9133fff700 1 -- 172.21.15.154:0/230482188 wait complete. 2023-11-15T05:06:11.620 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:12.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:11.616600+0000 mon.smithi086 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.154:0/230482188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:13.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:13.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:12.842053+0000 mgr.smithi086.hkfamo (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:14.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.173+0000 7fac7446e700 1 -- 172.21.15.154:0/3877161087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 msgr2=0x7fac6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:14.175 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.173+0000 7fac7446e700 1 --2- 172.21.15.154:0/3877161087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c0f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fac5c009fb0 tx=0x7fac5c009b30).stop 2023-11-15T05:06:14.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- 172.21.15.154:0/3877161087 shutdown_connections 2023-11-15T05:06:14.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 --2- 172.21.15.154:0/3877161087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c0f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:14.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- 172.21.15.154:0/3877161087 >> 172.21.15.154:0/3877161087 conn(0x7fac6c0f0b70 msgr2=0x7fac6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:14.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- 172.21.15.154:0/3877161087 shutdown_connections 2023-11-15T05:06:14.176 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- 172.21.15.154:0/3877161087 wait complete. 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 Processor -- start 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- start start 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.174+0000 7fac7446e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c107240 con 0x7fac6c0f5410 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7220a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:14.177 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7220a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40158/0 (socket says 172.21.15.154:40158) 2023-11-15T05:06:14.178 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7220a700 1 -- 172.21.15.154:0/2266546589 learned_addr learned my addr 172.21.15.154:0/2266546589 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:14.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7220a700 1 -- 172.21.15.154:0/2266546589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c005040 con 0x7fac6c0f5410 2023-11-15T05:06:14.179 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7220a700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fac5c013510 tx=0x7fac5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:14.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac5c009320 con 0x7fac6c0f5410 2023-11-15T05:06:14.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac5c013f00 con 0x7fac6c0f5410 2023-11-15T05:06:14.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c060220 con 0x7fac6c0f5410 2023-11-15T05:06:14.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac5c014d90 con 0x7fac6c0f5410 2023-11-15T05:06:14.180 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.175+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c10bd40 con 0x7fac6c0f5410 2023-11-15T05:06:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.176+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac5c0095f0 con 0x7fac6c0f5410 2023-11-15T05:06:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.176+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c045ad0 con 0x7fac6c0f5410 2023-11-15T05:06:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.176+0000 7fac62ffd700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 0x7fac58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.176+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac5c041000 con 0x7fac6c0f5410 2023-11-15T05:06:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.176+0000 7fac71a09700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 0x7fac58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:14.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.177+0000 7fac71a09700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 0x7fac58032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fac680096a0 tx=0x7fac68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:14.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.181+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac5c014420 con 0x7fac6c0f5410 2023-11-15T05:06:14.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.390+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac6c0fae60 con 0x7fac6c0f5410 2023-11-15T05:06:14.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.391+0000 7fac62ffd700 1 -- 172.21.15.154:0/2266546589 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac5c01d7f0 con 0x7fac6c0f5410 2023-11-15T05:06:14.392 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:14.392 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:14.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 msgr2=0x7fac58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:14.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 0x7fac58032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fac680096a0 tx=0x7fac68006b40).stop 2023-11-15T05:06:14.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 msgr2=0x7fac6c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:14.393 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fac5c013510 tx=0x7fac5c00b040).stop 2023-11-15T05:06:14.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 shutdown_connections 2023-11-15T05:06:14.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fac5802ff60 0x7fac58032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:14.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 --2- 172.21.15.154:0/2266546589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac6c0f5410 0x7fac6c10b850 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:14.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 >> 172.21.15.154:0/2266546589 conn(0x7fac6c0f0b70 msgr2=0x7fac6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:14.394 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 shutdown_connections 2023-11-15T05:06:14.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:14.392+0000 7fac7446e700 1 -- 172.21.15.154:0/2266546589 wait complete. 2023-11-15T05:06:14.395 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:14.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:14.391614+0000 mon.smithi086 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.154:0/2266546589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:15.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:14.842313+0000 mgr.smithi086.hkfamo (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:15.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:15.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:16.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 -- 172.21.15.154:0/4083315004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 msgr2=0x7fbff00f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:16.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 --2- 172.21.15.154:0/4083315004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff00f90d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fbfe0009fb0 tx=0x7fbfe0009b30).stop 2023-11-15T05:06:16.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 -- 172.21.15.154:0/4083315004 shutdown_connections 2023-11-15T05:06:16.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 --2- 172.21.15.154:0/4083315004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff00f90d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:16.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 -- 172.21.15.154:0/4083315004 >> 172.21.15.154:0/4083315004 conn(0x7fbff00f0b70 msgr2=0x7fbff00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:16.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 -- 172.21.15.154:0/4083315004 shutdown_connections 2023-11-15T05:06:16.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.865+0000 7fbff61f1700 1 -- 172.21.15.154:0/4083315004 wait complete. 2023-11-15T05:06:16.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.866+0000 7fbff61f1700 1 Processor -- start 2023-11-15T05:06:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.866+0000 7fbff61f1700 1 -- start start 2023-11-15T05:06:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.866+0000 7fbff61f1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.866+0000 7fbff61f1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff0106bc0 con 0x7fbff0060280 2023-11-15T05:06:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfef7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40180/0 (socket says 172.21.15.154:40180) 2023-11-15T05:06:16.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfef7fe700 1 -- 172.21.15.154:0/426945748 learned_addr learned my addr 172.21.15.154:0/426945748 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:16.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfef7fe700 1 -- 172.21.15.154:0/426945748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0005040 con 0x7fbff0060280 2023-11-15T05:06:16.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfef7fe700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fbfe0013510 tx=0x7fbfe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:16.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfe0009320 con 0x7fbff0060280 2023-11-15T05:06:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff0106dc0 con 0x7fbff0060280 2023-11-15T05:06:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfe0013f00 con 0x7fbff0060280 2023-11-15T05:06:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfe0014dc0 con 0x7fbff0060280 2023-11-15T05:06:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.867+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff0107250 con 0x7fbff0060280 2023-11-15T05:06:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.868+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfe0014420 con 0x7fbff0060280 2023-11-15T05:06:16.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.868+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff0045ad0 con 0x7fbff0060280 2023-11-15T05:06:16.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.868+0000 7fbfe7fff700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 0x7fbfdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:16.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.868+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfe0041110 con 0x7fbff0060280 2023-11-15T05:06:16.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.869+0000 7fbfeeffd700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 0x7fbfdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:16.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.869+0000 7fbfeeffd700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 0x7fbfdc032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbfd80096a0 tx=0x7fbfd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:16.875 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:16.873+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfe00095f0 con 0x7fbff0060280 2023-11-15T05:06:17.081 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.080+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbff0108100 con 0x7fbff0060280 2023-11-15T05:06:17.082 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.080+0000 7fbfe7fff700 1 -- 172.21.15.154:0/426945748 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfe001cb10 con 0x7fbff0060280 2023-11-15T05:06:17.083 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:17.083 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:17.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 msgr2=0x7fbfdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:17.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 0x7fbfdc032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbfd80096a0 tx=0x7fbfd8006b40).stop 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 msgr2=0x7fbff01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fbfe0013510 tx=0x7fbfe000b040).stop 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 shutdown_connections 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbfdc02ff60 0x7fbfdc032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 --2- 172.21.15.154:0/426945748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbff0060280 0x7fbff01066d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:17.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 >> 172.21.15.154:0/426945748 conn(0x7fbff00f0b70 msgr2=0x7fbff00f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:17.086 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 shutdown_connections 2023-11-15T05:06:17.086 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:17.082+0000 7fbff61f1700 1 -- 172.21.15.154:0/426945748 wait complete. 2023-11-15T05:06:17.086 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:16.842635+0000 mgr.smithi086.hkfamo (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:17.081570+0000 mon.smithi086 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.154:0/426945748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:18.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:18.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:18.842948+0000 mgr.smithi086.hkfamo (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:19.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 -- 172.21.15.154:0/685577090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 msgr2=0x7f4e980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:19.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/685577090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e980f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f4e8c009fb0 tx=0x7f4e8c009b30).stop 2023-11-15T05:06:19.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 -- 172.21.15.154:0/685577090 shutdown_connections 2023-11-15T05:06:19.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/685577090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e980f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:19.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 -- 172.21.15.154:0/685577090 >> 172.21.15.154:0/685577090 conn(0x7f4e980f0b70 msgr2=0x7f4e980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:19.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 -- 172.21.15.154:0/685577090 shutdown_connections 2023-11-15T05:06:19.709 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.701+0000 7f4e9ef43700 1 -- 172.21.15.154:0/685577090 wait complete. 2023-11-15T05:06:19.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ef43700 1 Processor -- start 2023-11-15T05:06:19.710 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ef43700 1 -- start start 2023-11-15T05:06:19.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ef43700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:19.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ef43700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98106c20 con 0x7f4e980f5410 2023-11-15T05:06:19.711 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ccdf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:19.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ccdf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40198/0 (socket says 172.21.15.154:40198) 2023-11-15T05:06:19.712 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.702+0000 7f4e9ccdf700 1 -- 172.21.15.154:0/2627184195 learned_addr learned my addr 172.21.15.154:0/2627184195 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:19.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e9ccdf700 1 -- 172.21.15.154:0/2627184195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c005040 con 0x7f4e980f5410 2023-11-15T05:06:19.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e9ccdf700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4e8c013510 tx=0x7f4e8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:19.713 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e8c009320 con 0x7f4e980f5410 2023-11-15T05:06:19.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e98106e20 con 0x7f4e980f5410 2023-11-15T05:06:19.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e8c013f00 con 0x7f4e980f5410 2023-11-15T05:06:19.714 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e8c014d90 con 0x7f4e980f5410 2023-11-15T05:06:19.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.703+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e981072b0 con 0x7f4e980f5410 2023-11-15T05:06:19.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e8c0095f0 con 0x7f4e980f5410 2023-11-15T05:06:19.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e98045ad0 con 0x7f4e980f5410 2023-11-15T05:06:19.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e957fa700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 0x7f4e88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:19.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e8c041000 con 0x7f4e980f5410 2023-11-15T05:06:19.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e97fff700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 0x7f4e88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:19.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.704+0000 7f4e97fff700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 0x7f4e88032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4e840096a0 tx=0x7f4e84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:19.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.708+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e8c014420 con 0x7f4e980f5410 2023-11-15T05:06:19.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.911+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e980fae60 con 0x7f4e980f5410 2023-11-15T05:06:19.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.912+0000 7f4e957fa700 1 -- 172.21.15.154:0/2627184195 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e8c010230 con 0x7f4e980f5410 2023-11-15T05:06:19.913 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:19.913 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:19.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 msgr2=0x7f4e88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:19.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 0x7f4e88032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4e840096a0 tx=0x7f4e84006b40).stop 2023-11-15T05:06:19.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 msgr2=0x7f4e98106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:19.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4e8c013510 tx=0x7f4e8c00b040).stop 2023-11-15T05:06:19.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 shutdown_connections 2023-11-15T05:06:19.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4e8802ff60 0x7f4e88032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:19.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 --2- 172.21.15.154:0/2627184195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e980f5410 0x7f4e98106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:19.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.913+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 >> 172.21.15.154:0/2627184195 conn(0x7f4e980f0b70 msgr2=0x7f4e980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:19.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.914+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 shutdown_connections 2023-11-15T05:06:19.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:19.914+0000 7f4e9ef43700 1 -- 172.21.15.154:0/2627184195 wait complete. 2023-11-15T05:06:19.917 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:20.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:19.912823+0000 mon.smithi086 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.154:0/2627184195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:20.843244+0000 mgr.smithi086.hkfamo (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:21.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:21.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:22.476 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- 172.21.15.154:0/2328532919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 msgr2=0x7faf8c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:22.477 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 --2- 172.21.15.154:0/2328532919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c0f7d00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7faf88009fb0 tx=0x7faf88009b30).stop 2023-11-15T05:06:22.477 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- 172.21.15.154:0/2328532919 shutdown_connections 2023-11-15T05:06:22.477 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 --2- 172.21.15.154:0/2328532919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c0f7d00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:22.477 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- 172.21.15.154:0/2328532919 >> 172.21.15.154:0/2328532919 conn(0x7faf8c0f0120 msgr2=0x7faf8c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:22.477 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- 172.21.15.154:0/2328532919 shutdown_connections 2023-11-15T05:06:22.478 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- 172.21.15.154:0/2328532919 wait complete. 2023-11-15T05:06:22.478 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 Processor -- start 2023-11-15T05:06:22.478 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- start start 2023-11-15T05:06:22.478 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:22.478 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.471+0000 7faf92b17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c1069d0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.479 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf91b15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:22.479 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf91b15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40212/0 (socket says 172.21.15.154:40212) 2023-11-15T05:06:22.479 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf91b15700 1 -- 172.21.15.154:0/2081135584 learned_addr learned my addr 172.21.15.154:0/2081135584 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:22.479 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf91b15700 1 -- 172.21.15.154:0/2081135584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88005040 con 0x7faf8c0f78e0 2023-11-15T05:06:22.479 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf91b15700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7faf88009830 tx=0x7faf8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf88009320 con 0x7faf8c0f78e0 2023-11-15T05:06:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf880136c0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c106bd0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf8801ee90 con 0x7faf8c0f78e0 2023-11-15T05:06:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.472+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c107060 con 0x7faf8c0f78e0 2023-11-15T05:06:22.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.473+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf8801e420 con 0x7faf8c0f78e0 2023-11-15T05:06:22.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.473+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c045ad0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.473+0000 7faf827fc700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 0x7faf780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:22.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.473+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf88040fa0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.474+0000 7faf91314700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 0x7faf780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:22.482 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.474+0000 7faf91314700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 0x7faf780323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:22.482 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.478+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf880095f0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.685 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.684+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf8c107f10 con 0x7faf8c0f78e0 2023-11-15T05:06:22.685 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.685+0000 7faf827fc700 1 -- 172.21.15.154:0/2081135584 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf8802c2d0 con 0x7faf8c0f78e0 2023-11-15T05:06:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:22.686 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:22.687 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 msgr2=0x7faf780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:22.687 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 0x7faf780323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).stop 2023-11-15T05:06:22.687 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 msgr2=0x7faf8c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:22.687 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7faf88009830 tx=0x7faf8800b040).stop 2023-11-15T05:06:22.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 shutdown_connections 2023-11-15T05:06:22.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf7802ff10 0x7faf780323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:22.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 --2- 172.21.15.154:0/2081135584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf8c0f78e0 0x7faf8c1064e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:22.688 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 >> 172.21.15.154:0/2081135584 conn(0x7faf8c0f0120 msgr2=0x7faf8c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:22.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 shutdown_connections 2023-11-15T05:06:22.689 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:22.686+0000 7faf92b17700 1 -- 172.21.15.154:0/2081135584 wait complete. 2023-11-15T05:06:22.689 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:22.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:22.685843+0000 mon.smithi086 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.154:0/2081135584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:23.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:22.843557+0000 mgr.smithi086.hkfamo (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:24.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:24.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:25.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/2042315942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 msgr2=0x7f8f780f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:25.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/2042315942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f780f7d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f8f68009fb0 tx=0x7f8f68009b30).stop 2023-11-15T05:06:25.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/2042315942 shutdown_connections 2023-11-15T05:06:25.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/2042315942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f780f7d00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:25.181 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/2042315942 >> 172.21.15.154:0/2042315942 conn(0x7f8f780f0120 msgr2=0x7f8f780f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/2042315942 shutdown_connections 2023-11-15T05:06:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/2042315942 wait complete. 2023-11-15T05:06:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.179+0000 7f8f7d3c2700 1 Processor -- start 2023-11-15T05:06:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f7d3c2700 1 -- start start 2023-11-15T05:06:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f7d3c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f7d3c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f781069d0 con 0x7f8f780f78e0 2023-11-15T05:06:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f77fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f77fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51874/0 (socket says 172.21.15.154:51874) 2023-11-15T05:06:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f77fff700 1 -- 172.21.15.154:0/1767184849 learned_addr learned my addr 172.21.15.154:0/1767184849 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f77fff700 1 -- 172.21.15.154:0/1767184849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68005040 con 0x7f8f780f78e0 2023-11-15T05:06:25.184 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f77fff700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8f68013510 tx=0x7f8f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:25.185 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.180+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f68009320 con 0x7f8f780f78e0 2023-11-15T05:06:25.185 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78106bd0 con 0x7f8f780f78e0 2023-11-15T05:06:25.185 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f68013f00 con 0x7f8f780f78e0 2023-11-15T05:06:25.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f68014dc0 con 0x7f8f780f78e0 2023-11-15T05:06:25.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f78107060 con 0x7f8f780f78e0 2023-11-15T05:06:25.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f68014420 con 0x7f8f780f78e0 2023-11-15T05:06:25.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.181+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78045ad0 con 0x7f8f780f78e0 2023-11-15T05:06:25.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.182+0000 7f8f74ff9700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:25.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.182+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f68040fd0 con 0x7f8f780f78e0 2023-11-15T05:06:25.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.182+0000 7f8f777fe700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:25.187 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.182+0000 7f8f777fe700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 0x7f8f60032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8f6c0096a0 tx=0x7f8f6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:25.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.186+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f6806d2b0 con 0x7f8f780f78e0 2023-11-15T05:06:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:24.843862+0000 mgr.smithi086.hkfamo (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:25.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.394+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f78107f10 con 0x7f8f780f78e0 2023-11-15T05:06:25.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.395+0000 7f8f74ff9700 1 -- 172.21.15.154:0/1767184849 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f680095f0 con 0x7f8f780f78e0 2023-11-15T05:06:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:25.396 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:25.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 msgr2=0x7f8f60032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:25.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 0x7f8f60032470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8f6c0096a0 tx=0x7f8f6c006b40).stop 2023-11-15T05:06:25.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 msgr2=0x7f8f781064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:25.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8f68013510 tx=0x7f8f6800b040).stop 2023-11-15T05:06:25.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 shutdown_connections 2023-11-15T05:06:25.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:25.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 --2- 172.21.15.154:0/1767184849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f780f78e0 0x7f8f781064e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:25.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.396+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 >> 172.21.15.154:0/1767184849 conn(0x7f8f780f0120 msgr2=0x7f8f780f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:25.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.397+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 shutdown_connections 2023-11-15T05:06:25.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:25.397+0000 7f8f7d3c2700 1 -- 172.21.15.154:0/1767184849 wait complete. 2023-11-15T05:06:25.400 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:24.929660+0000 mon.smithi086 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:06:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:24.936709+0000 mon.smithi086 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:06:26.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:25.395797+0000 mon.smithi086 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.154:0/1767184849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:26.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:26.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:26.844177+0000 mgr.smithi086.hkfamo (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:28.022 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- 172.21.15.154:0/3211969010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 msgr2=0x7f7f6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:28.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 --2- 172.21.15.154:0/3211969010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c0f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7f5c009fb0 tx=0x7f7f5c009b30).stop 2023-11-15T05:06:28.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- 172.21.15.154:0/3211969010 shutdown_connections 2023-11-15T05:06:28.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 --2- 172.21.15.154:0/3211969010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c0f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:28.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- 172.21.15.154:0/3211969010 >> 172.21.15.154:0/3211969010 conn(0x7f7f6c0f09a0 msgr2=0x7f7f6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:28.023 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- 172.21.15.154:0/3211969010 shutdown_connections 2023-11-15T05:06:28.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- 172.21.15.154:0/3211969010 wait complete. 2023-11-15T05:06:28.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 Processor -- start 2023-11-15T05:06:28.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- start start 2023-11-15T05:06:28.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:28.024 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.021+0000 7f7f72984700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c1058a0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f71982700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:28.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f71982700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51894/0 (socket says 172.21.15.154:51894) 2023-11-15T05:06:28.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f71982700 1 -- 172.21.15.154:0/538504134 learned_addr learned my addr 172.21.15.154:0/538504134 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:28.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f71982700 1 -- 172.21.15.154:0/538504134 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c005040 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f71982700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7f5c013510 tx=0x7f7f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:28.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5c009320 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f5c013f00 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c105aa0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5c014dc0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.022+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c105f30 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.023+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f5c0095f0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.023+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c045ad0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.023+0000 7f7f627fc700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 0x7f7f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.023+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f5c041000 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.023+0000 7f7f71181700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 0x7f7f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:28.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.024+0000 7f7f71181700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 0x7f7f58032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7f680096a0 tx=0x7f7f68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:28.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.028+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f5c014420 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.232 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.232+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f6c106de0 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.233+0000 7f7f627fc700 1 -- 172.21.15.154:0/538504134 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f5c010230 con 0x7f7f6c0f51a0 2023-11-15T05:06:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:28.234 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:28.235 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 msgr2=0x7f7f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:28.235 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 0x7f7f58032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f7f680096a0 tx=0x7f7f68006b40).stop 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 msgr2=0x7f7f6c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7f5c013510 tx=0x7f7f5c00b040).stop 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 shutdown_connections 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7f5802ff60 0x7f7f58032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 --2- 172.21.15.154:0/538504134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6c0f51a0 0x7f7f6c1053b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:28.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 >> 172.21.15.154:0/538504134 conn(0x7f7f6c0f09a0 msgr2=0x7f7f6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:28.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 shutdown_connections 2023-11-15T05:06:28.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:28.234+0000 7f7f72984700 1 -- 172.21.15.154:0/538504134 wait complete. 2023-11-15T05:06:28.237 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:28.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:28.233625+0000 mon.smithi086 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.154:0/538504134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:29.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:29.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:29.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:28.844633+0000 mgr.smithi086.hkfamo (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:30.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 -- 172.21.15.154:0/3887766195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 msgr2=0x7f60ec0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:30.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/3887766195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec0f7d00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f60dc009fb0 tx=0x7f60dc009b30).stop 2023-11-15T05:06:30.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 -- 172.21.15.154:0/3887766195 shutdown_connections 2023-11-15T05:06:30.788 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/3887766195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec0f7d00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:30.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 -- 172.21.15.154:0/3887766195 >> 172.21.15.154:0/3887766195 conn(0x7f60ec0f0120 msgr2=0x7f60ec0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:30.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 -- 172.21.15.154:0/3887766195 shutdown_connections 2023-11-15T05:06:30.789 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 -- 172.21.15.154:0/3887766195 wait complete. 2023-11-15T05:06:30.791 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.787+0000 7f60f0ae7700 1 Processor -- start 2023-11-15T05:06:30.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60f0ae7700 1 -- start start 2023-11-15T05:06:30.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60f0ae7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:30.792 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60f0ae7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec1069d0 con 0x7f60ec0f78e0 2023-11-15T05:06:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51912/0 (socket says 172.21.15.154:51912) 2023-11-15T05:06:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60eb7fe700 1 -- 172.21.15.154:0/2897033409 learned_addr learned my addr 172.21.15.154:0/2897033409 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60eb7fe700 1 -- 172.21.15.154:0/2897033409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc005040 con 0x7f60ec0f78e0 2023-11-15T05:06:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.788+0000 7f60eb7fe700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f60dc013510 tx=0x7f60dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:30.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60dc009320 con 0x7f60ec0f78e0 2023-11-15T05:06:30.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec106bd0 con 0x7f60ec0f78e0 2023-11-15T05:06:30.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60dc013f00 con 0x7f60ec0f78e0 2023-11-15T05:06:30.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60dc014dc0 con 0x7f60ec0f78e0 2023-11-15T05:06:30.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec107060 con 0x7f60ec0f78e0 2023-11-15T05:06:30.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60dc01d3d0 con 0x7f60ec0f78e0 2023-11-15T05:06:30.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.789+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec045ad0 con 0x7f60ec0f78e0 2023-11-15T05:06:30.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.790+0000 7f60d3fff700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 0x7f60d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:30.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.790+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60dc041400 con 0x7f60ec0f78e0 2023-11-15T05:06:30.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.790+0000 7f60eaffd700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 0x7f60d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:30.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.790+0000 7f60eaffd700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 0x7f60d4032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f60e00096a0 tx=0x7f60e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:30.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:30.794+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60dc0126f0 con 0x7f60ec0f78e0 2023-11-15T05:06:31.002 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.001+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60ec107f10 con 0x7f60ec0f78e0 2023-11-15T05:06:31.003 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.002+0000 7f60d3fff700 1 -- 172.21.15.154:0/2897033409 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60dc012330 con 0x7f60ec0f78e0 2023-11-15T05:06:31.003 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:31.003 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:31.005 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 msgr2=0x7f60d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:31.005 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 0x7f60d4032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f60e00096a0 tx=0x7f60e0006b40).stop 2023-11-15T05:06:31.005 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 msgr2=0x7f60ec1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:31.005 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f60dc013510 tx=0x7f60dc00b040).stop 2023-11-15T05:06:31.006 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 shutdown_connections 2023-11-15T05:06:31.006 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60d402ff60 0x7f60d4032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:31.006 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 --2- 172.21.15.154:0/2897033409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60ec0f78e0 0x7f60ec1064e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:31.007 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 >> 172.21.15.154:0/2897033409 conn(0x7f60ec0f0120 msgr2=0x7f60ec0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:31.007 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 shutdown_connections 2023-11-15T05:06:31.007 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:31.003+0000 7f60f0ae7700 1 -- 172.21.15.154:0/2897033409 wait complete. 2023-11-15T05:06:31.007 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:30.844973+0000 mgr.smithi086.hkfamo (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:32.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:31.002614+0000 mon.smithi086 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.154:0/2897033409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:32.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:32.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:32.845245+0000 mgr.smithi086.hkfamo (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:33.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 -- 172.21.15.154:0/2902740106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 msgr2=0x7fab0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:33.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 --2- 172.21.15.154:0/2902740106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7faaf4009fb0 tx=0x7faaf4009b30).stop 2023-11-15T05:06:33.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 -- 172.21.15.154:0/2902740106 shutdown_connections 2023-11-15T05:06:33.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 --2- 172.21.15.154:0/2902740106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:33.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 -- 172.21.15.154:0/2902740106 >> 172.21.15.154:0/2902740106 conn(0x7fab0c0f0b70 msgr2=0x7fab0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:33.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 -- 172.21.15.154:0/2902740106 shutdown_connections 2023-11-15T05:06:33.557 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.555+0000 7fab11512700 1 -- 172.21.15.154:0/2902740106 wait complete. 2023-11-15T05:06:33.559 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab11512700 1 Processor -- start 2023-11-15T05:06:33.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab11512700 1 -- start start 2023-11-15T05:06:33.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab11512700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:33.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab11512700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c060240 con 0x7fab0c0f5410 2023-11-15T05:06:33.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab0affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:33.560 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab0affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45624/0 (socket says 172.21.15.154:45624) 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.556+0000 7fab0affd700 1 -- 172.21.15.154:0/1665929635 learned_addr learned my addr 172.21.15.154:0/1665929635 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab0affd700 1 -- 172.21.15.154:0/1665929635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf4005040 con 0x7fab0c0f5410 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab0affd700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7faaf4013510 tx=0x7faaf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaf4009320 con 0x7fab0c0f5410 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c060440 con 0x7fab0c0f5410 2023-11-15T05:06:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faaf4013f00 con 0x7fab0c0f5410 2023-11-15T05:06:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaf4014d90 con 0x7fab0c0f5410 2023-11-15T05:06:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.557+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c109a80 con 0x7fab0c0f5410 2023-11-15T05:06:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faaf40263a0 con 0x7fab0c0f5410 2023-11-15T05:06:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c045ad0 con 0x7fab0c0f5410 2023-11-15T05:06:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab037fe700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 0x7faaf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faaf40410e0 con 0x7fab0c0f5410 2023-11-15T05:06:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab0a7fc700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 0x7faaf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.558+0000 7fab0a7fc700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 0x7faaf8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7faafc0096a0 tx=0x7faafc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.563+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faaf40126f0 con 0x7fab0c0f5410 2023-11-15T05:06:33.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.771+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab0c10a2a0 con 0x7fab0c0f5410 2023-11-15T05:06:33.772 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.772+0000 7fab037fe700 1 -- 172.21.15.154:0/1665929635 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faaf4012330 con 0x7fab0c0f5410 2023-11-15T05:06:33.773 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:33.773 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:33.774 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 msgr2=0x7faaf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:33.774 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 0x7faaf8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7faafc0096a0 tx=0x7faafc006b40).stop 2023-11-15T05:06:33.774 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 msgr2=0x7fab0c1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7faaf4013510 tx=0x7faaf400b040).stop 2023-11-15T05:06:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 shutdown_connections 2023-11-15T05:06:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faaf802ff60 0x7faaf8032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 --2- 172.21.15.154:0/1665929635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab0c0f5410 0x7fab0c1093d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 >> 172.21.15.154:0/1665929635 conn(0x7fab0c0f0b70 msgr2=0x7fab0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:33.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 shutdown_connections 2023-11-15T05:06:33.776 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:33.773+0000 7fab11512700 1 -- 172.21.15.154:0/1665929635 wait complete. 2023-11-15T05:06:33.776 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:34.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:33.772873+0000 mon.smithi086 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.154:0/1665929635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:35.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:34.845643+0000 mgr.smithi086.hkfamo (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:35.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:35.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:36.395 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- 172.21.15.154:0/497898819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 msgr2=0x7f1e8c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:36.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 --2- 172.21.15.154:0/497898819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c0f55c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1e88009fb0 tx=0x7f1e88009b30).stop 2023-11-15T05:06:36.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- 172.21.15.154:0/497898819 shutdown_connections 2023-11-15T05:06:36.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 --2- 172.21.15.154:0/497898819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c0f55c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:36.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- 172.21.15.154:0/497898819 >> 172.21.15.154:0/497898819 conn(0x7f1e8c0f09a0 msgr2=0x7f1e8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:36.396 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- 172.21.15.154:0/497898819 shutdown_connections 2023-11-15T05:06:36.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- 172.21.15.154:0/497898819 wait complete. 2023-11-15T05:06:36.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 Processor -- start 2023-11-15T05:06:36.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.394+0000 7f1e934e1700 1 -- start start 2023-11-15T05:06:36.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e934e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:36.397 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e934e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c1058d0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e924df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:36.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e924df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45650/0 (socket says 172.21.15.154:45650) 2023-11-15T05:06:36.398 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e924df700 1 -- 172.21.15.154:0/3974284314 learned_addr learned my addr 172.21.15.154:0/3974284314 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:36.399 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e924df700 1 -- 172.21.15.154:0/3974284314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88005040 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.395+0000 7f1e924df700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1e88013510 tx=0x7f1e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:36.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e88009320 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c105ad0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e88013f00 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.400 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e88014dc0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c105f60 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.396+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e880095f0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.397+0000 7f1e82ffd700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 0x7f1e78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.397+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c045ad0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.397+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e88041000 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.397+0000 7f1e91cde700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 0x7f1e78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:36.402 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.397+0000 7f1e91cde700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 0x7f1e78032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:36.402 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.402+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e8806d2b0 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.608 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.607+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e8c106e10 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.609 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.608+0000 7f1e82ffd700 1 -- 172.21.15.154:0/3974284314 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e88010230 con 0x7f1e8c0f51a0 2023-11-15T05:06:36.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:36.609 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:36.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.609+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 msgr2=0x7f1e78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:36.610 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.609+0000 7f1e934e1700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 0x7f1e78032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1e7c0096a0 tx=0x7f1e7c006b40).stop 2023-11-15T05:06:36.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.609+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 msgr2=0x7f1e8c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:36.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.609+0000 7f1e934e1700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1e88013510 tx=0x7f1e8800b040).stop 2023-11-15T05:06:36.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.609+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 shutdown_connections 2023-11-15T05:06:36.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.610+0000 7f1e934e1700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1e7802ff60 0x7f1e78032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:36.611 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.610+0000 7f1e934e1700 1 --2- 172.21.15.154:0/3974284314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e8c0f51a0 0x7f1e8c1053e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:36.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.610+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 >> 172.21.15.154:0/3974284314 conn(0x7f1e8c0f09a0 msgr2=0x7f1e8c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:36.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.610+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 shutdown_connections 2023-11-15T05:06:36.612 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:36.610+0000 7f1e934e1700 1 -- 172.21.15.154:0/3974284314 wait complete. 2023-11-15T05:06:36.612 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:36.608769+0000 mon.smithi086 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.154:0/3974284314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:37.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:36.846020+0000 mgr.smithi086.hkfamo (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:38.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:38.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:39.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 -- 172.21.15.154:0/3412605304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 msgr2=0x7fb8d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:39.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/3412605304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fb8c8009fb0 tx=0x7fb8c8009b30).stop 2023-11-15T05:06:39.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 -- 172.21.15.154:0/3412605304 shutdown_connections 2023-11-15T05:06:39.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/3412605304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:39.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 -- 172.21.15.154:0/3412605304 >> 172.21.15.154:0/3412605304 conn(0x7fb8d80f0b70 msgr2=0x7fb8d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:39.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.221+0000 7fb8e00dc700 1 -- 172.21.15.154:0/3412605304 shutdown_connections 2023-11-15T05:06:39.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.222+0000 7fb8e00dc700 1 -- 172.21.15.154:0/3412605304 wait complete. 2023-11-15T05:06:39.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.222+0000 7fb8e00dc700 1 Processor -- start 2023-11-15T05:06:39.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.222+0000 7fb8e00dc700 1 -- start start 2023-11-15T05:06:39.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.222+0000 7fb8e00dc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:39.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.222+0000 7fb8e00dc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d80fb820 con 0x7fb8d80f5410 2023-11-15T05:06:39.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8dde78700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:39.226 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8dde78700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45668/0 (socket says 172.21.15.154:45668) 2023-11-15T05:06:39.226 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8dde78700 1 -- 172.21.15.154:0/2499057131 learned_addr learned my addr 172.21.15.154:0/2499057131 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:39.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8dde78700 1 -- 172.21.15.154:0/2499057131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8005040 con 0x7fb8d80f5410 2023-11-15T05:06:39.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8dde78700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fb8c8013510 tx=0x7fb8c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:39.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c8009320 con 0x7fb8d80f5410 2023-11-15T05:06:39.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d80fb9c0 con 0x7fb8d80f5410 2023-11-15T05:06:39.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8c8013f00 con 0x7fb8d80f5410 2023-11-15T05:06:39.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c8014d90 con 0x7fb8d80f5410 2023-11-15T05:06:39.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.223+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d80fbe50 con 0x7fb8d80f5410 2023-11-15T05:06:39.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.224+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8c80095f0 con 0x7fb8d80f5410 2023-11-15T05:06:39.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.224+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d8059100 con 0x7fb8d80f5410 2023-11-15T05:06:39.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.224+0000 7fb8ce7fc700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 0x7fb8c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:39.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.224+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8c8041000 con 0x7fb8d80f5410 2023-11-15T05:06:39.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.225+0000 7fb8dd677700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 0x7fb8c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:39.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.225+0000 7fb8dd677700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 0x7fb8c4032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:39.231 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.229+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c8014420 con 0x7fb8d80f5410 2023-11-15T05:06:39.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:38.846421+0000 mgr.smithi086.hkfamo (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:39.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.436+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8d80fae60 con 0x7fb8d80f5410 2023-11-15T05:06:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.436+0000 7fb8ce7fc700 1 -- 172.21.15.154:0/2499057131 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8c8010230 con 0x7fb8d80f5410 2023-11-15T05:06:39.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:39.438 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 msgr2=0x7fb8c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 0x7fb8c4032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).stop 2023-11-15T05:06:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 msgr2=0x7fb8d80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fb8c8013510 tx=0x7fb8c800b040).stop 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 shutdown_connections 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb8c402ff60 0x7fb8c4032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 --2- 172.21.15.154:0/2499057131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d80f5410 0x7fb8d80fd3a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 >> 172.21.15.154:0/2499057131 conn(0x7fb8d80f0b70 msgr2=0x7fb8d805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 shutdown_connections 2023-11-15T05:06:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:39.438+0000 7fb8e00dc700 1 -- 172.21.15.154:0/2499057131 wait complete. 2023-11-15T05:06:39.441 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:40.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:39.437319+0000 mon.smithi086 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.154:0/2499057131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:40.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:40.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:40.846804+0000 mgr.smithi086.hkfamo (mgr.14162) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:42.036 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 -- 172.21.15.154:0/3897097243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 msgr2=0x7f83ec0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:42.037 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 --2- 172.21.15.154:0/3897097243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f4b60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f83dc009fb0 tx=0x7f83dc009b30).stop 2023-11-15T05:06:42.037 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 -- 172.21.15.154:0/3897097243 shutdown_connections 2023-11-15T05:06:42.037 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 --2- 172.21.15.154:0/3897097243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f4b60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 -- 172.21.15.154:0/3897097243 >> 172.21.15.154:0/3897097243 conn(0x7f83ec0f0260 msgr2=0x7f83ec0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.034+0000 7f83f243a700 1 -- 172.21.15.154:0/3897097243 shutdown_connections 2023-11-15T05:06:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.035+0000 7f83f243a700 1 -- 172.21.15.154:0/3897097243 wait complete. 2023-11-15T05:06:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.035+0000 7f83f243a700 1 Processor -- start 2023-11-15T05:06:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.035+0000 7f83f243a700 1 -- start start 2023-11-15T05:06:42.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83f243a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:42.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83f243a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ec0f5770 con 0x7f83ec0f4740 2023-11-15T05:06:42.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83ebfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:42.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83ebfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45692/0 (socket says 172.21.15.154:45692) 2023-11-15T05:06:42.039 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83ebfff700 1 -- 172.21.15.154:0/1932669598 learned_addr learned my addr 172.21.15.154:0/1932669598 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:42.040 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83ebfff700 1 -- 172.21.15.154:0/1932669598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc005040 con 0x7f83ec0f4740 2023-11-15T05:06:42.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.036+0000 7f83ebfff700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f83dc014000 tx=0x7f83dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:42.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83dc0095f0 con 0x7f83ec0f4740 2023-11-15T05:06:42.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ec0f5910 con 0x7f83ec0f4740 2023-11-15T05:06:42.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83dc007e70 con 0x7f83ec0f4740 2023-11-15T05:06:42.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83dc014ea0 con 0x7f83ec0f4740 2023-11-15T05:06:42.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ec0f5da0 con 0x7f83ec0f4740 2023-11-15T05:06:42.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83dc025070 con 0x7f83ec0f4740 2023-11-15T05:06:42.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.037+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ec059100 con 0x7f83ec0f4740 2023-11-15T05:06:42.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.038+0000 7f83e8ff9700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:42.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.038+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83dc040d60 con 0x7f83ec0f4740 2023-11-15T05:06:42.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.038+0000 7f83eb7fe700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:42.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.038+0000 7f83eb7fe700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 0x7f83d4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f83e00096a0 tx=0x7f83e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:42.044 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.042+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83dc045070 con 0x7f83ec0f4740 2023-11-15T05:06:42.247 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.246+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83ec045ad0 con 0x7f83ec0f4740 2023-11-15T05:06:42.247 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83e8ff9700 1 -- 172.21.15.154:0/1932669598 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83dc00f710 con 0x7f83ec0f4740 2023-11-15T05:06:42.248 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:42.248 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:42.249 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 msgr2=0x7f83d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:42.249 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 0x7f83d4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f83e00096a0 tx=0x7f83e0006b40).stop 2023-11-15T05:06:42.249 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 msgr2=0x7f83ec0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f83dc014000 tx=0x7f83dc00b040).stop 2023-11-15T05:06:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 shutdown_connections 2023-11-15T05:06:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 --2- 172.21.15.154:0/1932669598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec0f4740 0x7f83ec0f72f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.248+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 >> 172.21.15.154:0/1932669598 conn(0x7f83ec0f0260 msgr2=0x7f83ec0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:42.251 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.249+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 shutdown_connections 2023-11-15T05:06:42.251 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:42.249+0000 7f83f243a700 1 -- 172.21.15.154:0/1932669598 wait complete. 2023-11-15T05:06:42.251 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:42.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:42.247872+0000 mon.smithi086 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.154:0/1932669598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:43.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:43.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:43.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:42.847192+0000 mgr.smithi086.hkfamo (mgr.14162) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:44.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 -- 172.21.15.154:0/1638849801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 msgr2=0x7f4ca00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:44.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 --2- 172.21.15.154:0/1638849801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca00f7d00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f4c9c009fb0 tx=0x7f4c9c009b30).stop 2023-11-15T05:06:44.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 -- 172.21.15.154:0/1638849801 shutdown_connections 2023-11-15T05:06:44.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 --2- 172.21.15.154:0/1638849801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca00f7d00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:44.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 -- 172.21.15.154:0/1638849801 >> 172.21.15.154:0/1638849801 conn(0x7f4ca00f0120 msgr2=0x7f4ca00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:44.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 -- 172.21.15.154:0/1638849801 shutdown_connections 2023-11-15T05:06:44.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 -- 172.21.15.154:0/1638849801 wait complete. 2023-11-15T05:06:44.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.812+0000 7f4ca70be700 1 Processor -- start 2023-11-15T05:06:44.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca70be700 1 -- start start 2023-11-15T05:06:44.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca70be700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:44.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca70be700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0108bd0 con 0x7f4ca00f78e0 2023-11-15T05:06:44.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca60bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:44.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca60bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43830/0 (socket says 172.21.15.154:43830) 2023-11-15T05:06:44.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca60bc700 1 -- 172.21.15.154:0/768408029 learned_addr learned my addr 172.21.15.154:0/768408029 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:44.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca60bc700 1 -- 172.21.15.154:0/768408029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c005040 con 0x7f4ca00f78e0 2023-11-15T05:06:44.817 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca60bc700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4c9c013510 tx=0x7f4c9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:44.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c9c009320 con 0x7f4ca00f78e0 2023-11-15T05:06:44.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0108dd0 con 0x7f4ca00f78e0 2023-11-15T05:06:44.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c9c013f00 con 0x7f4ca00f78e0 2023-11-15T05:06:44.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.813+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c9c014dc0 con 0x7f4ca00f78e0 2023-11-15T05:06:44.821 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.814+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca0109260 con 0x7f4ca00f78e0 2023-11-15T05:06:44.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.814+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c9c014420 con 0x7f4ca00f78e0 2023-11-15T05:06:44.822 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.814+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0045ad0 con 0x7f4ca00f78e0 2023-11-15T05:06:44.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.814+0000 7f4c96ffd700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 0x7f4c8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:44.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.814+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c9c041030 con 0x7f4ca00f78e0 2023-11-15T05:06:44.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.815+0000 7f4ca58bb700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 0x7f4c8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:44.823 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.815+0000 7f4ca58bb700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 0x7f4c8c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:44.824 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:44.819+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c9c0095f0 con 0x7f4ca00f78e0 2023-11-15T05:06:45.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.026+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca010a110 con 0x7f4ca00f78e0 2023-11-15T05:06:45.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.027+0000 7f4c96ffd700 1 -- 172.21.15.154:0/768408029 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c9c01cb10 con 0x7f4ca00f78e0 2023-11-15T05:06:45.028 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:45.028 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:45.030 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.028+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 msgr2=0x7f4c8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:45.030 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 0x7f4c8c032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f4c900096a0 tx=0x7f4c90006b40).stop 2023-11-15T05:06:45.030 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 msgr2=0x7f4ca01086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:45.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4c9c013510 tx=0x7f4c9c00b040).stop 2023-11-15T05:06:45.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 shutdown_connections 2023-11-15T05:06:45.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4c8c02ff60 0x7f4c8c032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:45.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 --2- 172.21.15.154:0/768408029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca00f78e0 0x7f4ca01086e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:45.031 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 >> 172.21.15.154:0/768408029 conn(0x7f4ca00f0120 msgr2=0x7f4ca00f0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:45.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 shutdown_connections 2023-11-15T05:06:45.032 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:45.029+0000 7f4ca70be700 1 -- 172.21.15.154:0/768408029 wait complete. 2023-11-15T05:06:45.032 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:44.847623+0000 mgr.smithi086.hkfamo (mgr.14162) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:46.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:45.028079+0000 mon.smithi086 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.154:0/768408029' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:46.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:46.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:47.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- 172.21.15.154:0/2001478993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 msgr2=0x7f41640f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:47.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 --2- 172.21.15.154:0/2001478993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f8fc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f4160009fb0 tx=0x7f4160009b30).stop 2023-11-15T05:06:47.567 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- 172.21.15.154:0/2001478993 shutdown_connections 2023-11-15T05:06:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 --2- 172.21.15.154:0/2001478993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f8fc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- 172.21.15.154:0/2001478993 >> 172.21.15.154:0/2001478993 conn(0x7f41640f09a0 msgr2=0x7f41640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- 172.21.15.154:0/2001478993 shutdown_connections 2023-11-15T05:06:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- 172.21.15.154:0/2001478993 wait complete. 2023-11-15T05:06:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 Processor -- start 2023-11-15T05:06:47.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.565+0000 7f416ad6d700 1 -- start start 2023-11-15T05:06:47.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f416ad6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:47.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f416ad6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41640f4ec0 con 0x7f4164060140 2023-11-15T05:06:47.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f4169d6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:47.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f4169d6b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43842/0 (socket says 172.21.15.154:43842) 2023-11-15T05:06:47.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f4169d6b700 1 -- 172.21.15.154:0/3996778482 learned_addr learned my addr 172.21.15.154:0/3996778482 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:47.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f4169d6b700 1 -- 172.21.15.154:0/3996778482 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4160005040 con 0x7f4164060140 2023-11-15T05:06:47.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f4169d6b700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4160013510 tx=0x7f416000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:47.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.566+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4160009320 con 0x7f4164060140 2023-11-15T05:06:47.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41640f5060 con 0x7f4164060140 2023-11-15T05:06:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4160013f00 con 0x7f4164060140 2023-11-15T05:06:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4160014d90 con 0x7f4164060140 2023-11-15T05:06:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41640f54f0 con 0x7f4164060140 2023-11-15T05:06:47.573 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41600095f0 con 0x7f4164060140 2023-11-15T05:06:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4164059100 con 0x7f4164060140 2023-11-15T05:06:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.567+0000 7f415a7fc700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:47.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.568+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4160041000 con 0x7f4164060140 2023-11-15T05:06:47.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.568+0000 7f416956a700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:47.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.568+0000 7f416956a700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 0x7f4150032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f41540096a0 tx=0x7f4154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:47.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.572+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4160014420 con 0x7f4164060140 2023-11-15T05:06:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:46.848016+0000 mgr.smithi086.hkfamo (mgr.14162) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.777+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41640fbce0 con 0x7f4164060140 2023-11-15T05:06:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.777+0000 7f415a7fc700 1 -- 172.21.15.154:0/3996778482 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f416001d7f0 con 0x7f4164060140 2023-11-15T05:06:47.779 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:47.779 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 msgr2=0x7f4150032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 0x7f4150032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f41540096a0 tx=0x7f4154006b40).stop 2023-11-15T05:06:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 msgr2=0x7f41640f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f4160013510 tx=0x7f416000b040).stop 2023-11-15T05:06:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 shutdown_connections 2023-11-15T05:06:47.782 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:47.782 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 --2- 172.21.15.154:0/3996778482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4164060140 0x7f41640f6a40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:47.782 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 >> 172.21.15.154:0/3996778482 conn(0x7f41640f09a0 msgr2=0x7f41640f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:47.783 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 shutdown_connections 2023-11-15T05:06:47.783 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:47.779+0000 7f416ad6d700 1 -- 172.21.15.154:0/3996778482 wait complete. 2023-11-15T05:06:47.783 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:48.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:47.779501+0000 mon.smithi086 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.154:0/3996778482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:49.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:49.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:49.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:48.848336+0000 mgr.smithi086.hkfamo (mgr.14162) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:50.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 -- 172.21.15.154:0/1827053119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 msgr2=0x7faa340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:50.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 --2- 172.21.15.154:0/1827053119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa340f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7faa28009fb0 tx=0x7faa28009b30).stop 2023-11-15T05:06:50.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 -- 172.21.15.154:0/1827053119 shutdown_connections 2023-11-15T05:06:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 --2- 172.21.15.154:0/1827053119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa340f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 -- 172.21.15.154:0/1827053119 >> 172.21.15.154:0/1827053119 conn(0x7faa340f0b70 msgr2=0x7faa340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 -- 172.21.15.154:0/1827053119 shutdown_connections 2023-11-15T05:06:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.293+0000 7faa3adcb700 1 -- 172.21.15.154:0/1827053119 wait complete. 2023-11-15T05:06:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.294+0000 7faa3adcb700 1 Processor -- start 2023-11-15T05:06:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.294+0000 7faa3adcb700 1 -- start start 2023-11-15T05:06:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.294+0000 7faa3adcb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.294+0000 7faa3adcb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa34060240 con 0x7faa340f5410 2023-11-15T05:06:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa38b67700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa38b67700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43852/0 (socket says 172.21.15.154:43852) 2023-11-15T05:06:50.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa38b67700 1 -- 172.21.15.154:0/248686019 learned_addr learned my addr 172.21.15.154:0/248686019 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:50.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa38b67700 1 -- 172.21.15.154:0/248686019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28005040 con 0x7faa340f5410 2023-11-15T05:06:50.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa38b67700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7faa28013510 tx=0x7faa2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:50.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa28009320 con 0x7faa340f5410 2023-11-15T05:06:50.300 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34060440 con 0x7faa340f5410 2023-11-15T05:06:50.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa28013f00 con 0x7faa340f5410 2023-11-15T05:06:50.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa28014d90 con 0x7faa340f5410 2023-11-15T05:06:50.301 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.295+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa34109ae0 con 0x7faa340f5410 2023-11-15T05:06:50.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.296+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa280263a0 con 0x7faa340f5410 2023-11-15T05:06:50.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.296+0000 7faa317fa700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 0x7faa24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:50.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.296+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa280410b0 con 0x7faa340f5410 2023-11-15T05:06:50.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.296+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa34045ad0 con 0x7faa340f5410 2023-11-15T05:06:50.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.296+0000 7faa33fff700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 0x7faa24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:50.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.297+0000 7faa33fff700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 0x7faa24032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7faa200096a0 tx=0x7faa20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:50.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.302+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa28012920 con 0x7faa340f5410 2023-11-15T05:06:50.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.507+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa3410a230 con 0x7faa340f5410 2023-11-15T05:06:50.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.508+0000 7faa317fa700 1 -- 172.21.15.154:0/248686019 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa2802b020 con 0x7faa340f5410 2023-11-15T05:06:50.511 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:50.511 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:50.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 msgr2=0x7faa24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:50.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 0x7faa24032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7faa200096a0 tx=0x7faa20006b40).stop 2023-11-15T05:06:50.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 msgr2=0x7faa34109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:50.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7faa28013510 tx=0x7faa2800b040).stop 2023-11-15T05:06:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 shutdown_connections 2023-11-15T05:06:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faa2402ff60 0x7faa24032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 --2- 172.21.15.154:0/248686019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa340f5410 0x7faa34109430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 >> 172.21.15.154:0/248686019 conn(0x7faa340f0b70 msgr2=0x7faa340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 shutdown_connections 2023-11-15T05:06:50.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:50.510+0000 7faa3adcb700 1 -- 172.21.15.154:0/248686019 wait complete. 2023-11-15T05:06:50.514 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:50.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:50.509246+0000 mon.smithi086 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.154:0/248686019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:51.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:51.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:51.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:50.848674+0000 mgr.smithi086.hkfamo (mgr.14162) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:53.053 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 -- 172.21.15.154:0/713823460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 msgr2=0x7f12ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:53.053 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 --2- 172.21.15.154:0/713823460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f12a0009fb0 tx=0x7f12a0009b30).stop 2023-11-15T05:06:53.053 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 -- 172.21.15.154:0/713823460 shutdown_connections 2023-11-15T05:06:53.053 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 --2- 172.21.15.154:0/713823460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 -- 172.21.15.154:0/713823460 >> 172.21.15.154:0/713823460 conn(0x7f12ac0f0b70 msgr2=0x7f12ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 -- 172.21.15.154:0/713823460 shutdown_connections 2023-11-15T05:06:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.051+0000 7f12b1e84700 1 -- 172.21.15.154:0/713823460 wait complete. 2023-11-15T05:06:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12b1e84700 1 Processor -- start 2023-11-15T05:06:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12b1e84700 1 -- start start 2023-11-15T05:06:53.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12b1e84700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:53.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12b1e84700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac107210 con 0x7f12ac0f5410 2023-11-15T05:06:53.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12ab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:53.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12ab7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43858/0 (socket says 172.21.15.154:43858) 2023-11-15T05:06:53.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.052+0000 7f12ab7fe700 1 -- 172.21.15.154:0/1419066275 learned_addr learned my addr 172.21.15.154:0/1419066275 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:53.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f12ab7fe700 1 -- 172.21.15.154:0/1419066275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0005040 con 0x7f12ac0f5410 2023-11-15T05:06:53.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f12ab7fe700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f12a0013510 tx=0x7f12a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:53.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a0009320 con 0x7f12ac0f5410 2023-11-15T05:06:53.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac060220 con 0x7f12ac0f5410 2023-11-15T05:06:53.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12a0013f00 con 0x7f12ac0f5410 2023-11-15T05:06:53.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a0014d90 con 0x7f12ac0f5410 2023-11-15T05:06:53.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.053+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac10bd10 con 0x7f12ac0f5410 2023-11-15T05:06:53.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.054+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12a00095f0 con 0x7f12ac0f5410 2023-11-15T05:06:53.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.054+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac045ad0 con 0x7f12ac0f5410 2023-11-15T05:06:53.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.054+0000 7f1293fff700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 0x7f1294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:53.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.054+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12a0040fe0 con 0x7f12ac0f5410 2023-11-15T05:06:53.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.054+0000 7f12aaffd700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 0x7f1294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:53.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.055+0000 7f12aaffd700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 0x7f1294032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f129c0096a0 tx=0x7f129c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:53.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.059+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12a001d7f0 con 0x7f12ac0f5410 2023-11-15T05:06:53.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:52.848906+0000 mgr.smithi086.hkfamo (mgr.14162) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:53.267 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.265+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12ac0fae60 con 0x7f12ac0f5410 2023-11-15T05:06:53.267 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.266+0000 7f1293fff700 1 -- 172.21.15.154:0/1419066275 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12a0010230 con 0x7f12ac0f5410 2023-11-15T05:06:53.267 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:53.267 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:53.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 msgr2=0x7f1294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:53.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 0x7f1294032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f129c0096a0 tx=0x7f129c006b40).stop 2023-11-15T05:06:53.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 msgr2=0x7f12ac10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:53.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f12a0013510 tx=0x7f12a000b040).stop 2023-11-15T05:06:53.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 shutdown_connections 2023-11-15T05:06:53.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f129402ff60 0x7f1294032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:53.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 --2- 172.21.15.154:0/1419066275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac0f5410 0x7f12ac10b820 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:53.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 >> 172.21.15.154:0/1419066275 conn(0x7f12ac0f0b70 msgr2=0x7f12ac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:53.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 shutdown_connections 2023-11-15T05:06:53.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:53.267+0000 7f12b1e84700 1 -- 172.21.15.154:0/1419066275 wait complete. 2023-11-15T05:06:53.270 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:53.266692+0000 mon.smithi086 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.154:0/1419066275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:54.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:54.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:54.849156+0000 mgr.smithi086.hkfamo (mgr.14162) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:55.895 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- 172.21.15.154:0/3940796767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 msgr2=0x7fc1c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:55.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/3940796767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c00f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fc1b0009fb0 tx=0x7fc1b0009b30).stop 2023-11-15T05:06:55.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- 172.21.15.154:0/3940796767 shutdown_connections 2023-11-15T05:06:55.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/3940796767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c00f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:55.896 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- 172.21.15.154:0/3940796767 >> 172.21.15.154:0/3940796767 conn(0x7fc1c00f0b70 msgr2=0x7fc1c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:55.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- 172.21.15.154:0/3940796767 shutdown_connections 2023-11-15T05:06:55.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- 172.21.15.154:0/3940796767 wait complete. 2023-11-15T05:06:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 Processor -- start 2023-11-15T05:06:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.895+0000 7fc1c7c95700 1 -- start start 2023-11-15T05:06:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c7c95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c7c95700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0105b20 con 0x7fc1c00f5410 2023-11-15T05:06:55.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c5a31700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:55.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c5a31700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:46472/0 (socket says 172.21.15.154:46472) 2023-11-15T05:06:55.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c5a31700 1 -- 172.21.15.154:0/1743239181 learned_addr learned my addr 172.21.15.154:0/1743239181 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:55.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c5a31700 1 -- 172.21.15.154:0/1743239181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0005040 con 0x7fc1c00f5410 2023-11-15T05:06:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c5a31700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc1b0013510 tx=0x7fc1b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b0009320 con 0x7fc1c00f5410 2023-11-15T05:06:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c0105d20 con 0x7fc1c00f5410 2023-11-15T05:06:55.901 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.896+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1b0013f00 con 0x7fc1c00f5410 2023-11-15T05:06:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.897+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b0014d90 con 0x7fc1c00f5410 2023-11-15T05:06:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.897+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c01061b0 con 0x7fc1c00f5410 2023-11-15T05:06:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.897+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1b00095f0 con 0x7fc1c00f5410 2023-11-15T05:06:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.897+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0045ad0 con 0x7fc1c00f5410 2023-11-15T05:06:55.902 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.897+0000 7fc1b67fc700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 0x7fc1ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:55.903 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.898+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1b0042e30 con 0x7fc1c00f5410 2023-11-15T05:06:55.903 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.898+0000 7fc1c5230700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 0x7fc1ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:55.903 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.898+0000 7fc1c5230700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 0x7fc1ac032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:55.904 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:55.902+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1b0014420 con 0x7fc1c00f5410 2023-11-15T05:06:56.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.107+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1c00fae60 con 0x7fc1c00f5410 2023-11-15T05:06:56.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.107+0000 7fc1b67fc700 1 -- 172.21.15.154:0/1743239181 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1b0011290 con 0x7fc1c00f5410 2023-11-15T05:06:56.108 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:56.108 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:56.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 msgr2=0x7fc1ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:56.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 0x7fc1ac032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).stop 2023-11-15T05:06:56.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 msgr2=0x7fc1c0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:56.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc1b0013510 tx=0x7fc1b000b040).stop 2023-11-15T05:06:56.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 shutdown_connections 2023-11-15T05:06:56.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc1ac02ff60 0x7fc1ac032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:56.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 --2- 172.21.15.154:0/1743239181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1c00f5410 0x7fc1c0105630 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:56.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 >> 172.21.15.154:0/1743239181 conn(0x7fc1c00f0b70 msgr2=0x7fc1c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:56.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 shutdown_connections 2023-11-15T05:06:56.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:56.109+0000 7fc1c7c95700 1 -- 172.21.15.154:0/1743239181 wait complete. 2023-11-15T05:06:56.112 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:56.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:56.108428+0000 mon.smithi086 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.154:0/1743239181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:06:57.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:06:57.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:06:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:56.849440+0000 mgr.smithi086.hkfamo (mgr.14162) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:06:58.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 -- 172.21.15.154:0/1137019429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 msgr2=0x7fc97c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:58.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 --2- 172.21.15.154:0/1137019429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c0f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc970009fb0 tx=0x7fc970009b30).stop 2023-11-15T05:06:58.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 -- 172.21.15.154:0/1137019429 shutdown_connections 2023-11-15T05:06:58.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 --2- 172.21.15.154:0/1137019429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c0f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:58.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 -- 172.21.15.154:0/1137019429 >> 172.21.15.154:0/1137019429 conn(0x7fc97c0f0b70 msgr2=0x7fc97c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:58.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 -- 172.21.15.154:0/1137019429 shutdown_connections 2023-11-15T05:06:58.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.716+0000 7fc98303d700 1 -- 172.21.15.154:0/1137019429 wait complete. 2023-11-15T05:06:58.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc98303d700 1 Processor -- start 2023-11-15T05:06:58.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc98303d700 1 -- start start 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc98303d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc98303d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c105b20 con 0x7fc97c0f5410 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc980dd9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc980dd9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:46490/0 (socket says 172.21.15.154:46490) 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.717+0000 7fc980dd9700 1 -- 172.21.15.154:0/983388124 learned_addr learned my addr 172.21.15.154:0/983388124 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:06:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc980dd9700 1 -- 172.21.15.154:0/983388124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc970005040 con 0x7fc97c0f5410 2023-11-15T05:06:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc980dd9700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc970013510 tx=0x7fc97000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc970009320 con 0x7fc97c0f5410 2023-11-15T05:06:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc970013f00 con 0x7fc97c0f5410 2023-11-15T05:06:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c105d20 con 0x7fc97c0f5410 2023-11-15T05:06:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc970014d90 con 0x7fc97c0f5410 2023-11-15T05:06:58.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.718+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c1061b0 con 0x7fc97c0f5410 2023-11-15T05:06:58.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9700263a0 con 0x7fc97c0f5410 2023-11-15T05:06:58.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc9797fa700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 0x7fc96c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:06:58.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc970041fb0 con 0x7fc97c0f5410 2023-11-15T05:06:58.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c045ad0 con 0x7fc97c0f5410 2023-11-15T05:06:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc97bfff700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 0x7fc96c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:06:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.719+0000 7fc97bfff700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 0x7fc96c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc9680096a0 tx=0x7fc968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:06:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.724+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9700682b0 con 0x7fc97c0f5410 2023-11-15T05:06:58.930 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.929+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc97c0faf50 con 0x7fc97c0f5410 2023-11-15T05:06:58.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.930+0000 7fc9797fa700 1 -- 172.21.15.154:0/983388124 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc97002b020 con 0x7fc97c0f5410 2023-11-15T05:06:58.931 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:06:58.931 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:06:58.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 msgr2=0x7fc96c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:58.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 0x7fc96c032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc9680096a0 tx=0x7fc968006b40).stop 2023-11-15T05:06:58.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 msgr2=0x7fc97c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:06:58.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc970013510 tx=0x7fc97000b040).stop 2023-11-15T05:06:58.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 shutdown_connections 2023-11-15T05:06:58.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc96c02ff60 0x7fc96c032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:58.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 --2- 172.21.15.154:0/983388124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c0f5410 0x7fc97c105630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:06:58.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 >> 172.21.15.154:0/983388124 conn(0x7fc97c0f0b70 msgr2=0x7fc97c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:06:58.935 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 shutdown_connections 2023-11-15T05:06:58.936 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:06:58.931+0000 7fc98303d700 1 -- 172.21.15.154:0/983388124 wait complete. 2023-11-15T05:06:58.937 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:06:59.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:06:58.849702+0000 mgr.smithi086.hkfamo (mgr.14162) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:00.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:06:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:06:58.930840+0000 mon.smithi086 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.154:0/983388124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:00.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:00.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:00.849897+0000 mgr.smithi086.hkfamo (mgr.14162) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:01.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.435+0000 7fc7b2808700 1 -- 172.21.15.154:0/2211136905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 msgr2=0x7fc7ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:01.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.435+0000 7fc7b2808700 1 --2- 172.21.15.154:0/2211136905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac0f55c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fc79c009fb0 tx=0x7fc79c009b30).stop 2023-11-15T05:07:01.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.435+0000 7fc7b2808700 1 -- 172.21.15.154:0/2211136905 shutdown_connections 2023-11-15T05:07:01.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.435+0000 7fc7b2808700 1 --2- 172.21.15.154:0/2211136905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac0f55c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:01.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.435+0000 7fc7b2808700 1 -- 172.21.15.154:0/2211136905 >> 172.21.15.154:0/2211136905 conn(0x7fc7ac0f09a0 msgr2=0x7fc7ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:01.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 -- 172.21.15.154:0/2211136905 shutdown_connections 2023-11-15T05:07:01.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 -- 172.21.15.154:0/2211136905 wait complete. 2023-11-15T05:07:01.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 Processor -- start 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 -- start start 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b2808700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac1069d0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b1806700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b1806700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:46510/0 (socket says 172.21.15.154:46510) 2023-11-15T05:07:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.436+0000 7fc7b1806700 1 -- 172.21.15.154:0/447095429 learned_addr learned my addr 172.21.15.154:0/447095429 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:01.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7b1806700 1 -- 172.21.15.154:0/447095429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c005040 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7b1806700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc79c009830 tx=0x7fc79c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:01.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc79c009320 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac106bd0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc79c0136c0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc79c01ee90 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.437+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac107060 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.438+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac045ad0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.443 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.438+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc79c01e420 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.443 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.438+0000 7fc7a27fc700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 0x7fc7980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:01.443 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.438+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc79c0410b0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.443 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.438+0000 7fc7b1005700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 0x7fc7980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:01.443 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.439+0000 7fc7b1005700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 0x7fc7980323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc7a80096a0 tx=0x7fc7a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:01.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.442+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc79c06d2b0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.651+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7ac107f10 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.651+0000 7fc7a27fc700 1 -- 172.21.15.154:0/447095429 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc79c02c2d0 con 0x7fc7ac0f51a0 2023-11-15T05:07:01.652 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:01.653 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:01.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 msgr2=0x7fc7980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:01.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 0x7fc7980323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc7a80096a0 tx=0x7fc7a8006b40).stop 2023-11-15T05:07:01.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 msgr2=0x7fc7ac1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:01.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc79c009830 tx=0x7fc79c00b040).stop 2023-11-15T05:07:01.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 shutdown_connections 2023-11-15T05:07:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc79802ff10 0x7fc7980323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 --2- 172.21.15.154:0/447095429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac0f51a0 0x7fc7ac1064e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 >> 172.21.15.154:0/447095429 conn(0x7fc7ac0f09a0 msgr2=0x7fc7ac05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 shutdown_connections 2023-11-15T05:07:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:01.653+0000 7fc7b2808700 1 -- 172.21.15.154:0/447095429 wait complete. 2023-11-15T05:07:01.656 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:02.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:01.652396+0000 mon.smithi086 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.154:0/447095429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:03.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:03.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:03.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:02.850179+0000 mgr.smithi086.hkfamo (mgr.14162) 170 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:04.232 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 -- 172.21.15.154:0/1925981549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 msgr2=0x7f8ac40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:04.232 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 --2- 172.21.15.154:0/1925981549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac40f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f8ab8009fb0 tx=0x7f8ab8009b30).stop 2023-11-15T05:07:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 -- 172.21.15.154:0/1925981549 shutdown_connections 2023-11-15T05:07:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 --2- 172.21.15.154:0/1925981549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac40f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 -- 172.21.15.154:0/1925981549 >> 172.21.15.154:0/1925981549 conn(0x7f8ac40f0b70 msgr2=0x7f8ac40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 -- 172.21.15.154:0/1925981549 shutdown_connections 2023-11-15T05:07:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.231+0000 7f8ac9937700 1 -- 172.21.15.154:0/1925981549 wait complete. 2023-11-15T05:07:04.235 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.232+0000 7f8ac9937700 1 Processor -- start 2023-11-15T05:07:04.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.232+0000 7f8ac9937700 1 -- start start 2023-11-15T05:07:04.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.232+0000 7f8ac9937700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:04.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.232+0000 7f8ac9937700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4107210 con 0x7f8ac40f5410 2023-11-15T05:07:04.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac2ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:04.236 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac2ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38400/0 (socket says 172.21.15.154:38400) 2023-11-15T05:07:04.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac2ffd700 1 -- 172.21.15.154:0/2926211751 learned_addr learned my addr 172.21.15.154:0/2926211751 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:04.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac2ffd700 1 -- 172.21.15.154:0/2926211751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8005040 con 0x7f8ac40f5410 2023-11-15T05:07:04.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac2ffd700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8ab8013510 tx=0x7f8ab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:04.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab8009320 con 0x7f8ac40f5410 2023-11-15T05:07:04.237 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac4060220 con 0x7f8ac40f5410 2023-11-15T05:07:04.238 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ab8013f00 con 0x7f8ac40f5410 2023-11-15T05:07:04.238 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab8014d90 con 0x7f8ac40f5410 2023-11-15T05:07:04.238 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.233+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac410bd10 con 0x7f8ac40f5410 2023-11-15T05:07:04.239 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.234+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ab8014420 con 0x7f8ac40f5410 2023-11-15T05:07:04.239 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.234+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac4045ad0 con 0x7f8ac40f5410 2023-11-15T05:07:04.239 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.234+0000 7f8aab7fe700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 0x7f8aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:04.239 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.234+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ab80410c0 con 0x7f8ac40f5410 2023-11-15T05:07:04.239 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.234+0000 7f8ac27fc700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 0x7f8aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:04.240 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.235+0000 7f8ac27fc700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 0x7f8aac032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8ab40096a0 tx=0x7f8ab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:04.240 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.239+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ab80095f0 con 0x7f8ac40f5410 2023-11-15T05:07:04.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.444+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ac40fae60 con 0x7f8ac40f5410 2023-11-15T05:07:04.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.444+0000 7f8aab7fe700 1 -- 172.21.15.154:0/2926211751 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ab801db70 con 0x7f8ac40f5410 2023-11-15T05:07:04.445 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:04.445 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:04.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 msgr2=0x7f8aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:04.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 0x7f8aac032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8ab40096a0 tx=0x7f8ab4006b40).stop 2023-11-15T05:07:04.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 msgr2=0x7f8ac410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:04.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8ab8013510 tx=0x7f8ab800b040).stop 2023-11-15T05:07:04.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 shutdown_connections 2023-11-15T05:07:04.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f8aac02ff60 0x7f8aac032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:04.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 --2- 172.21.15.154:0/2926211751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac40f5410 0x7f8ac410b820 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:04.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 >> 172.21.15.154:0/2926211751 conn(0x7f8ac40f0b70 msgr2=0x7f8ac40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:04.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 shutdown_connections 2023-11-15T05:07:04.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:04.446+0000 7f8ac9937700 1 -- 172.21.15.154:0/2926211751 wait complete. 2023-11-15T05:07:04.450 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:04.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:04.445379+0000 mon.smithi086 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.154:0/2926211751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:05.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:05.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:05.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:04.850474+0000 mgr.smithi086.hkfamo (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:06.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.905+0000 7fb3030d5700 1 -- 172.21.15.154:0/3413786769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 msgr2=0x7fb2fc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:06.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.905+0000 7fb3030d5700 1 --2- 172.21.15.154:0/3413786769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc0f4b60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fb2f0009fb0 tx=0x7fb2f0009b30).stop 2023-11-15T05:07:06.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.905+0000 7fb3030d5700 1 -- 172.21.15.154:0/3413786769 shutdown_connections 2023-11-15T05:07:06.907 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.905+0000 7fb3030d5700 1 --2- 172.21.15.154:0/3413786769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc0f4b60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:06.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.905+0000 7fb3030d5700 1 -- 172.21.15.154:0/3413786769 >> 172.21.15.154:0/3413786769 conn(0x7fb2fc0f0260 msgr2=0x7fb2fc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:06.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.906+0000 7fb3030d5700 1 -- 172.21.15.154:0/3413786769 shutdown_connections 2023-11-15T05:07:06.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.906+0000 7fb3030d5700 1 -- 172.21.15.154:0/3413786769 wait complete. 2023-11-15T05:07:06.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.906+0000 7fb3030d5700 1 Processor -- start 2023-11-15T05:07:06.908 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.906+0000 7fb3030d5700 1 -- start start 2023-11-15T05:07:06.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb3030d5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:06.910 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb3030d5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc106bc0 con 0x7fb2fc0f4740 2023-11-15T05:07:06.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb300e71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:06.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb300e71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38424/0 (socket says 172.21.15.154:38424) 2023-11-15T05:07:06.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb300e71700 1 -- 172.21.15.154:0/490851417 learned_addr learned my addr 172.21.15.154:0/490851417 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:06.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb300e71700 1 -- 172.21.15.154:0/490851417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0005040 con 0x7fb2fc0f4740 2023-11-15T05:07:06.911 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb300e71700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb2f0013510 tx=0x7fb2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:06.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f0009320 con 0x7fb2fc0f4740 2023-11-15T05:07:06.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2f0013f00 con 0x7fb2fc0f4740 2023-11-15T05:07:06.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.907+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc106dc0 con 0x7fb2fc0f4740 2023-11-15T05:07:06.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f0014d90 con 0x7fb2fc0f4740 2023-11-15T05:07:06.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc107250 con 0x7fb2fc0f4740 2023-11-15T05:07:06.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2f00095f0 con 0x7fb2fc0f4740 2023-11-15T05:07:06.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc045ad0 con 0x7fb2fc0f4740 2023-11-15T05:07:06.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb2f97fa700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 0x7fb2ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:06.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.908+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2f0041000 con 0x7fb2fc0f4740 2023-11-15T05:07:06.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.909+0000 7fb2fbfff700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 0x7fb2ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:06.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.909+0000 7fb2fbfff700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 0x7fb2ec032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb2e80096a0 tx=0x7fb2e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:06.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:06.913+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2f001d7f0 con 0x7fb2fc0f4740 2023-11-15T05:07:07.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.121+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2fc108100 con 0x7fb2fc0f4740 2023-11-15T05:07:07.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.122+0000 7fb2f97fa700 1 -- 172.21.15.154:0/490851417 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2f0010230 con 0x7fb2fc0f4740 2023-11-15T05:07:07.123 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:07.123 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:07.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 msgr2=0x7fb2ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:07.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 0x7fb2ec032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb2e80096a0 tx=0x7fb2e8006b40).stop 2023-11-15T05:07:07.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 msgr2=0x7fb2fc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:07.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb2f0013510 tx=0x7fb2f000b040).stop 2023-11-15T05:07:07.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 shutdown_connections 2023-11-15T05:07:07.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fb2ec02ff60 0x7fb2ec032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:07.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 --2- 172.21.15.154:0/490851417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2fc0f4740 0x7fb2fc1066d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:07.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 >> 172.21.15.154:0/490851417 conn(0x7fb2fc0f0260 msgr2=0x7fb2fc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:07.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.123+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 shutdown_connections 2023-11-15T05:07:07.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:07.124+0000 7fb3030d5700 1 -- 172.21.15.154:0/490851417 wait complete. 2023-11-15T05:07:07.126 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:06.850730+0000 mgr.smithi086.hkfamo (mgr.14162) 172 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:07.122791+0000 mon.smithi086 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.154:0/490851417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:08.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:08.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:09.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:08.851096+0000 mgr.smithi086.hkfamo (mgr.14162) 173 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- 172.21.15.154:0/2585854698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 msgr2=0x7f7a480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2585854698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a480f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f7a3c009fb0 tx=0x7f7a3c009b30).stop 2023-11-15T05:07:09.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- 172.21.15.154:0/2585854698 shutdown_connections 2023-11-15T05:07:09.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2585854698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a480f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:09.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- 172.21.15.154:0/2585854698 >> 172.21.15.154:0/2585854698 conn(0x7f7a480f09a0 msgr2=0x7f7a480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:09.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- 172.21.15.154:0/2585854698 shutdown_connections 2023-11-15T05:07:09.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- 172.21.15.154:0/2585854698 wait complete. 2023-11-15T05:07:09.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 Processor -- start 2023-11-15T05:07:09.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.639+0000 7f7a4759e700 1 -- start start 2023-11-15T05:07:09.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4759e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:09.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4759e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a481058a0 con 0x7f7a480f51a0 2023-11-15T05:07:09.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:09.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38450/0 (socket says 172.21.15.154:38450) 2023-11-15T05:07:09.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4659c700 1 -- 172.21.15.154:0/2500887470 learned_addr learned my addr 172.21.15.154:0/2500887470 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:09.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4659c700 1 -- 172.21.15.154:0/2500887470 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c005040 con 0x7f7a480f51a0 2023-11-15T05:07:09.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.640+0000 7f7a4659c700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7a3c013510 tx=0x7f7a3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:09.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.641+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3c009320 con 0x7f7a480f51a0 2023-11-15T05:07:09.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.641+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a3c013f00 con 0x7f7a480f51a0 2023-11-15T05:07:09.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.641+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a48105aa0 con 0x7f7a480f51a0 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.641+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3c014dc0 con 0x7f7a480f51a0 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.641+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a48105f30 con 0x7f7a480f51a0 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a48045ad0 con 0x7f7a480f51a0 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a3c0095f0 con 0x7f7a480f51a0 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a36ffd700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 0x7f7a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:09.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a3c041040 con 0x7f7a480f51a0 2023-11-15T05:07:09.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a45d9b700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 0x7f7a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:09.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.642+0000 7f7a45d9b700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 0x7f7a30032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:09.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.647+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a3c014420 con 0x7f7a480f51a0 2023-11-15T05:07:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.856+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a48106de0 con 0x7f7a480f51a0 2023-11-15T05:07:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.857+0000 7f7a36ffd700 1 -- 172.21.15.154:0/2500887470 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a3c010230 con 0x7f7a480f51a0 2023-11-15T05:07:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:09.858 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 msgr2=0x7f7a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 0x7f7a30032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7a380096a0 tx=0x7f7a38006b40).stop 2023-11-15T05:07:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 msgr2=0x7f7a481053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f7a3c013510 tx=0x7f7a3c00b040).stop 2023-11-15T05:07:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 shutdown_connections 2023-11-15T05:07:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a3002ff60 0x7f7a30032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 --2- 172.21.15.154:0/2500887470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a480f51a0 0x7f7a481053b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:09.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 >> 172.21.15.154:0/2500887470 conn(0x7f7a480f09a0 msgr2=0x7f7a480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:09.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 shutdown_connections 2023-11-15T05:07:09.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:09.859+0000 7f7a4759e700 1 -- 172.21.15.154:0/2500887470 wait complete. 2023-11-15T05:07:09.863 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:10.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:09.857999+0000 mon.smithi086 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.154:0/2500887470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:11.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:11.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:10.851422+0000 mgr.smithi086.hkfamo (mgr.14162) 174 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:12.343 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- 172.21.15.154:0/148718057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 msgr2=0x7fcfcc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:12.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 --2- 172.21.15.154:0/148718057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc0f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fcfbc009fb0 tx=0x7fcfbc009b30).stop 2023-11-15T05:07:12.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- 172.21.15.154:0/148718057 shutdown_connections 2023-11-15T05:07:12.344 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 --2- 172.21.15.154:0/148718057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc0f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:12.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- 172.21.15.154:0/148718057 >> 172.21.15.154:0/148718057 conn(0x7fcfcc0f1c70 msgr2=0x7fcfcc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:12.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- 172.21.15.154:0/148718057 shutdown_connections 2023-11-15T05:07:12.345 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- 172.21.15.154:0/148718057 wait complete. 2023-11-15T05:07:12.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 Processor -- start 2023-11-15T05:07:12.346 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- start start 2023-11-15T05:07:12.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:12.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.339+0000 7fcfd4467700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc106c20 con 0x7fcfcc0f6510 2023-11-15T05:07:12.347 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd2203700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:12.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd2203700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:38458/0 (socket says 172.21.15.154:38458) 2023-11-15T05:07:12.348 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd2203700 1 -- 172.21.15.154:0/1856486031 learned_addr learned my addr 172.21.15.154:0/1856486031 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:12.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd2203700 1 -- 172.21.15.154:0/1856486031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc005040 con 0x7fcfcc0f6510 2023-11-15T05:07:12.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd2203700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcfbc014000 tx=0x7fcfbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:12.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfbc0095f0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.349 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc106e20 con 0x7fcfcc0f6510 2023-11-15T05:07:12.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfbc007e70 con 0x7fcfcc0f6510 2023-11-15T05:07:12.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfbc014ea0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.340+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc1072b0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.350 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.341+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc045ad0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.342+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfbc025070 con 0x7fcfcc0f6510 2023-11-15T05:07:12.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.342+0000 7fcfc2ffd700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 0x7fcfb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:12.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.342+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfbc040d10 con 0x7fcfcc0f6510 2023-11-15T05:07:12.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.342+0000 7fcfd1a02700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 0x7fcfb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:12.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.343+0000 7fcfd1a02700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 0x7fcfb8032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcfc80096a0 tx=0x7fcfc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:12.352 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.346+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfbc0256f0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.551+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfcc0fc050 con 0x7fcfcc0f6510 2023-11-15T05:07:12.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.552+0000 7fcfc2ffd700 1 -- 172.21.15.154:0/1856486031 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfbc0232d0 con 0x7fcfcc0f6510 2023-11-15T05:07:12.552 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:12.553 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:12.554 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 msgr2=0x7fcfb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:12.554 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 0x7fcfb8032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fcfc80096a0 tx=0x7fcfc8006b40).stop 2023-11-15T05:07:12.554 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 msgr2=0x7fcfcc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:12.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcfbc014000 tx=0x7fcfbc00b040).stop 2023-11-15T05:07:12.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 shutdown_connections 2023-11-15T05:07:12.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fcfb802ff60 0x7fcfb8032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:12.555 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 --2- 172.21.15.154:0/1856486031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfcc0f6510 0x7fcfcc106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:12.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 >> 172.21.15.154:0/1856486031 conn(0x7fcfcc0f1c70 msgr2=0x7fcfcc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:12.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 shutdown_connections 2023-11-15T05:07:12.556 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:12.553+0000 7fcfd4467700 1 -- 172.21.15.154:0/1856486031 wait complete. 2023-11-15T05:07:12.557 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:12.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:12.552528+0000 mon.smithi086 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.154:0/1856486031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:13.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:13.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:13.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:12.851673+0000 mgr.smithi086.hkfamo (mgr.14162) 175 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:15.089 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- 172.21.15.154:0/3470054259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 msgr2=0x7f72200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:15.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 --2- 172.21.15.154:0/3470054259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72200f66c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f721c009fb0 tx=0x7f721c009b30).stop 2023-11-15T05:07:15.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- 172.21.15.154:0/3470054259 shutdown_connections 2023-11-15T05:07:15.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 --2- 172.21.15.154:0/3470054259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72200f66c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:15.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- 172.21.15.154:0/3470054259 >> 172.21.15.154:0/3470054259 conn(0x7f72200f1aa0 msgr2=0x7f72200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:15.090 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- 172.21.15.154:0/3470054259 shutdown_connections 2023-11-15T05:07:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- 172.21.15.154:0/3470054259 wait complete. 2023-11-15T05:07:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 Processor -- start 2023-11-15T05:07:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.088+0000 7f7226409700 1 -- start start 2023-11-15T05:07:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7226409700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7226409700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72201069a0 con 0x7f72200f62a0 2023-11-15T05:07:15.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7225407700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:15.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7225407700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34162/0 (socket says 172.21.15.154:34162) 2023-11-15T05:07:15.092 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7225407700 1 -- 172.21.15.154:0/2398105783 learned_addr learned my addr 172.21.15.154:0/2398105783 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:15.093 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7225407700 1 -- 172.21.15.154:0/2398105783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c005040 con 0x7f72200f62a0 2023-11-15T05:07:15.094 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7225407700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f721c013510 tx=0x7f721c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:15.094 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c009320 con 0x7f72200f62a0 2023-11-15T05:07:15.094 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f721c013f00 con 0x7f72200f62a0 2023-11-15T05:07:15.094 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7220106ba0 con 0x7f72200f62a0 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.089+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c014dc0 con 0x7f72200f62a0 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.090+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7220107030 con 0x7f72200f62a0 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.090+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7220045ad0 con 0x7f72200f62a0 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.091+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f721c0095f0 con 0x7f72200f62a0 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.091+0000 7f7215ffb700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 0x7f720c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:15.095 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.091+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f721c041040 con 0x7f72200f62a0 2023-11-15T05:07:15.096 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.091+0000 7f7224c06700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 0x7f720c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:15.096 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.091+0000 7f7224c06700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 0x7f720c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f72100096a0 tx=0x7f7210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:15.096 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.095+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f721c014420 con 0x7f72200f62a0 2023-11-15T05:07:15.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:14.851922+0000 mgr.smithi086.hkfamo (mgr.14162) 176 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:15.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.303+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7220107ee0 con 0x7f72200f62a0 2023-11-15T05:07:15.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.303+0000 7f7215ffb700 1 -- 172.21.15.154:0/2398105783 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f721c010230 con 0x7f72200f62a0 2023-11-15T05:07:15.304 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:15.304 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:15.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 msgr2=0x7f720c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:15.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 0x7f720c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f72100096a0 tx=0x7f7210006b40).stop 2023-11-15T05:07:15.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 msgr2=0x7f72201064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:15.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f721c013510 tx=0x7f721c00b040).stop 2023-11-15T05:07:15.306 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 shutdown_connections 2023-11-15T05:07:15.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f720c02ff60 0x7f720c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:15.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 --2- 172.21.15.154:0/2398105783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200f62a0 0x7f72201064b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:15.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 >> 172.21.15.154:0/2398105783 conn(0x7f72200f1aa0 msgr2=0x7f72200f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:15.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 shutdown_connections 2023-11-15T05:07:15.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:15.305+0000 7f7226409700 1 -- 172.21.15.154:0/2398105783 wait complete. 2023-11-15T05:07:15.308 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:16.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:15.304397+0000 mon.smithi086 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.154:0/2398105783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:16.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:16.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:16.852169+0000 mgr.smithi086.hkfamo (mgr.14162) 177 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:17.864 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 -- 172.21.15.154:0/2568801403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 msgr2=0x7fc9ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:17.864 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/2568801403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac0f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc9a0009fb0 tx=0x7fc9a0009b30).stop 2023-11-15T05:07:17.864 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 -- 172.21.15.154:0/2568801403 shutdown_connections 2023-11-15T05:07:17.864 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/2568801403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac0f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:17.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 -- 172.21.15.154:0/2568801403 >> 172.21.15.154:0/2568801403 conn(0x7fc9ac0f0b70 msgr2=0x7fc9ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:17.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 -- 172.21.15.154:0/2568801403 shutdown_connections 2023-11-15T05:07:17.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.863+0000 7fc9b11e5700 1 -- 172.21.15.154:0/2568801403 wait complete. 2023-11-15T05:07:17.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9b11e5700 1 Processor -- start 2023-11-15T05:07:17.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9b11e5700 1 -- start start 2023-11-15T05:07:17.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9b11e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:17.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9b11e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac105b20 con 0x7fc9ac0f5410 2023-11-15T05:07:17.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9aad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:17.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9aad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34178/0 (socket says 172.21.15.154:34178) 2023-11-15T05:07:17.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.864+0000 7fc9aad9d700 1 -- 172.21.15.154:0/1602982961 learned_addr learned my addr 172.21.15.154:0/1602982961 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:17.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9aad9d700 1 -- 172.21.15.154:0/1602982961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0005040 con 0x7fc9ac0f5410 2023-11-15T05:07:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9aad9d700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc9a0013510 tx=0x7fc9a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a0009320 con 0x7fc9ac0f5410 2023-11-15T05:07:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac105d20 con 0x7fc9ac0f5410 2023-11-15T05:07:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9a0013f00 con 0x7fc9ac0f5410 2023-11-15T05:07:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a0014d90 con 0x7fc9ac0f5410 2023-11-15T05:07:17.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.865+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac1061b0 con 0x7fc9ac0f5410 2023-11-15T05:07:17.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.866+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9a00263a0 con 0x7fc9ac0f5410 2023-11-15T05:07:17.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.866+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac045ad0 con 0x7fc9ac0f5410 2023-11-15T05:07:17.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.866+0000 7fc9937fe700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 0x7fc994032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:17.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.866+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9a00413c0 con 0x7fc9ac0f5410 2023-11-15T05:07:17.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.866+0000 7fc9aa59c700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 0x7fc994032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:17.874 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.867+0000 7fc9aa59c700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 0x7fc994032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc99c0096a0 tx=0x7fc99c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:17.875 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:17.872+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9a001cb10 con 0x7fc9ac0f5410 2023-11-15T05:07:18.080 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.080+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9ac0fae60 con 0x7fc9ac0f5410 2023-11-15T05:07:18.081 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.080+0000 7fc9937fe700 1 -- 172.21.15.154:0/1602982961 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc9a001cb10 con 0x7fc9ac0f5410 2023-11-15T05:07:18.081 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:18.082 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:18.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 msgr2=0x7fc994032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:18.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 0x7fc994032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc99c0096a0 tx=0x7fc99c006b40).stop 2023-11-15T05:07:18.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 msgr2=0x7fc9ac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:18.083 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc9a0013510 tx=0x7fc9a000b040).stop 2023-11-15T05:07:18.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 shutdown_connections 2023-11-15T05:07:18.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc99402ff60 0x7fc994032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:18.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 --2- 172.21.15.154:0/1602982961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9ac0f5410 0x7fc9ac105630 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:18.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 >> 172.21.15.154:0/1602982961 conn(0x7fc9ac0f0b70 msgr2=0x7fc9ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:18.084 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 shutdown_connections 2023-11-15T05:07:18.085 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:18.082+0000 7fc9b11e5700 1 -- 172.21.15.154:0/1602982961 wait complete. 2023-11-15T05:07:18.085 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:18.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:07:17.861839+0000 mgr.smithi086.hkfamo (mgr.14162) 178 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi086 ... 2023-11-15T05:07:18.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:07:18.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:07:18.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:02:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:03:04Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:03:41Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.492 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:07:18.492 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:07:18.492 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:07:18.492 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:04:55Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.493 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:05:32Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.493 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:06:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.493 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:07:18.493 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:07:18.493 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:07:18.494 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: main() 2023-11-15T05:07:18.494 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:07:18.494 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: r = ctx.func(ctx) 2023-11-15T05:07:18.494 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:07:18.494 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: return func(ctx) 2023-11-15T05:07:18.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:07:18.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:07:18.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:07:18.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:07:18.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:07:18.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: raise RuntimeError('uid/gid not found') 2023-11-15T05:07:18.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: RuntimeError: uid/gid not found 2023-11-15T05:07:18.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:07:18.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T05:07:18.497 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: yield (conn, connr) 2023-11-15T05:07:18.497 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T05:07:18.497 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: code, '\n'.join(err))) 2023-11-15T05:07:18.497 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi086 ... 2023-11-15T05:07:18.497 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:07:18.498 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:07:18.498 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:02:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.498 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:03:04Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:03:41Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:07:18.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi086 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:07:18.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:07:18.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:04:55Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:05:32Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr time="2023-11-15T05:06:40Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:07:18.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:07:18.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:07:18.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:07:18.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: main() 2023-11-15T05:07:18.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:07:18.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: r = ctx.func(ctx) 2023-11-15T05:07:18.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:07:18.502 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: return func(ctx) 2023-11-15T05:07:18.502 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:07:18.502 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:07:18.502 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:07:18.502 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:07:18.503 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:07:18.503 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: raise RuntimeError('uid/gid not found') 2023-11-15T05:07:18.503 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: RuntimeError: uid/gid not found 2023-11-15T05:07:18.503 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:17.862985+0000 mon.smithi086 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T05:07:18.503 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:17.865371+0000 mon.smithi086 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T05:07:18.504 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:17.866171+0000 mon.smithi086 (mon.0) 244 : audit [DBG] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T05:07:18.504 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:18.081465+0000 mon.smithi086 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.154:0/1602982961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:19.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:19.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:19.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:07:17.866761+0000 mgr.smithi086.hkfamo (mgr.14162) 179 : cephadm [INF] Deploying daemon crash.smithi086 on smithi086 2023-11-15T05:07:19.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:18.852479+0000 mgr.smithi086.hkfamo (mgr.14162) 180 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:19.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.315675+0000 mon.smithi086 (mon.0) 246 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:07:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- 172.21.15.154:0/1347868112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 msgr2=0x7f2ffc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:20.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 --2- 172.21.15.154:0/1347868112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc0f66c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f2ff8009fb0 tx=0x7f2ff8009b30).stop 2023-11-15T05:07:20.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- 172.21.15.154:0/1347868112 shutdown_connections 2023-11-15T05:07:20.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 --2- 172.21.15.154:0/1347868112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc0f66c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:20.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- 172.21.15.154:0/1347868112 >> 172.21.15.154:0/1347868112 conn(0x7f2ffc0f1aa0 msgr2=0x7f2ffc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- 172.21.15.154:0/1347868112 shutdown_connections 2023-11-15T05:07:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- 172.21.15.154:0/1347868112 wait complete. 2023-11-15T05:07:20.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 Processor -- start 2023-11-15T05:07:20.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.578+0000 7f300244c700 1 -- start start 2023-11-15T05:07:20.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300244c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:20.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300244c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc1069a0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.582 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300144a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300144a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:34204/0 (socket says 172.21.15.154:34204) 2023-11-15T05:07:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300144a700 1 -- 172.21.15.154:0/1637155875 learned_addr learned my addr 172.21.15.154:0/1637155875 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300144a700 1 -- 172.21.15.154:0/1637155875 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff8005040 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300144a700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2ff8013510 tx=0x7f2ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff8009320 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.584 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ff8013f00 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.584 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff8014dc0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.584 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.579+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc106ba0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.584 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.580+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc107030 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.584 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.580+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ff80095f0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.585 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.580+0000 7f2ff1ffb700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 0x7f2fe8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:20.585 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.580+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc045ad0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.585 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.580+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ff80410b0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.585 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.581+0000 7f3000c49700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 0x7f2fe8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:20.586 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.581+0000 7f3000c49700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 0x7f2fe8032470 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f2fec0096a0 tx=0x7f2fec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:20.586 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.585+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ff801d7f0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.793+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ffc107ee0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.794+0000 7f2ff1ffb700 1 -- 172.21.15.154:0/1637155875 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ff801d7f0 con 0x7f2ffc0f62a0 2023-11-15T05:07:20.795 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:20.795 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:20.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 msgr2=0x7f2fe8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:20.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 0x7f2fe8032470 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f2fec0096a0 tx=0x7f2fec006b40).stop 2023-11-15T05:07:20.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 msgr2=0x7f2ffc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:20.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2ff8013510 tx=0x7f2ff800b040).stop 2023-11-15T05:07:20.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 shutdown_connections 2023-11-15T05:07:20.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2fe802ffb0 0x7f2fe8032470 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:20.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 --2- 172.21.15.154:0/1637155875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc0f62a0 0x7f2ffc1064b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:20.799 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 >> 172.21.15.154:0/1637155875 conn(0x7f2ffc0f1aa0 msgr2=0x7f2ffc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:20.799 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 shutdown_connections 2023-11-15T05:07:20.800 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:20.795+0000 7f300244c700 1 -- 172.21.15.154:0/1637155875 wait complete. 2023-11-15T05:07:20.800 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:20.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.582867+0000 mon.smithi086 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:07:20.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.586034+0000 mon.smithi086 (mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:07:20.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.586449+0000 mon.smithi086 (mon.0) 249 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:07:20.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.586861+0000 mgr.smithi086.hkfamo (mgr.14162) 181 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:07:20.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:19.589912+0000 mon.smithi086 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:07:20.990 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:07:19.592131+0000 mgr.smithi086.hkfamo (mgr.14162) 182 : cephadm [INF] Deploying daemon grafana.smithi086 on smithi086 2023-11-15T05:07:21.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:21 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:20.794830+0000 mon.smithi086 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.154:0/1637155875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:21.991 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:21 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:20.852730+0000 mgr.smithi086.hkfamo (mgr.14162) 183 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:22.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:22.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:23.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:22.852901+0000 mgr.smithi086.hkfamo (mgr.14162) 184 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:23.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 -- 172.21.15.154:0/1007286023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 msgr2=0x7ff9a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:23.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/1007286023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a40f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7ff98c009fb0 tx=0x7ff98c009b30).stop 2023-11-15T05:07:23.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 -- 172.21.15.154:0/1007286023 shutdown_connections 2023-11-15T05:07:23.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/1007286023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a40f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:23.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 -- 172.21.15.154:0/1007286023 >> 172.21.15.154:0/1007286023 conn(0x7ff9a40f1c70 msgr2=0x7ff9a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:23.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 -- 172.21.15.154:0/1007286023 shutdown_connections 2023-11-15T05:07:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 -- 172.21.15.154:0/1007286023 wait complete. 2023-11-15T05:07:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.310+0000 7ff9aa54f700 1 Processor -- start 2023-11-15T05:07:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9aa54f700 1 -- start start 2023-11-15T05:07:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9aa54f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9aa54f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4069e20 con 0x7ff9a40f6510 2023-11-15T05:07:23.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9a3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:23.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9a3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40890/0 (socket says 172.21.15.154:40890) 2023-11-15T05:07:23.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9a3fff700 1 -- 172.21.15.154:0/945446876 learned_addr learned my addr 172.21.15.154:0/945446876 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:23.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9a3fff700 1 -- 172.21.15.154:0/945446876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c005040 con 0x7ff9a40f6510 2023-11-15T05:07:23.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.311+0000 7ff9a3fff700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ff98c014000 tx=0x7ff98c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c0095f0 con 0x7ff9a40f6510 2023-11-15T05:07:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff98c007e70 con 0x7ff9a40f6510 2023-11-15T05:07:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a4069fc0 con 0x7ff9a40f6510 2023-11-15T05:07:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c014ea0 con 0x7ff9a40f6510 2023-11-15T05:07:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a406a450 con 0x7ff9a40f6510 2023-11-15T05:07:23.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff98c025070 con 0x7ff9a40f6510 2023-11-15T05:07:23.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.312+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a4059100 con 0x7ff9a40f6510 2023-11-15T05:07:23.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.313+0000 7ff9a0ff9700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:23.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.313+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff98c040970 con 0x7ff9a40f6510 2023-11-15T05:07:23.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.313+0000 7ff9a37fe700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:23.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.313+0000 7ff9a37fe700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff9940096a0 tx=0x7ff994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:23.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.318+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff98c0079c0 con 0x7ff9a40f6510 2023-11-15T05:07:23.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.524+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9a40fbf60 con 0x7ff9a40f6510 2023-11-15T05:07:23.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.524+0000 7ff9a0ff9700 1 -- 172.21.15.154:0/945446876 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff98c00f300 con 0x7ff9a40f6510 2023-11-15T05:07:23.526 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:23.527 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:23.527 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 msgr2=0x7ff990032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:23.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 0x7ff990032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff9940096a0 tx=0x7ff994006b40).stop 2023-11-15T05:07:23.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 msgr2=0x7ff9a406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:23.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7ff98c014000 tx=0x7ff98c00b040).stop 2023-11-15T05:07:23.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 shutdown_connections 2023-11-15T05:07:23.528 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff99002ff60 0x7ff990032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:23.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 --2- 172.21.15.154:0/945446876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a40f6510 0x7ff9a406b9a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:23.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 >> 172.21.15.154:0/945446876 conn(0x7ff9a40f1c70 msgr2=0x7ff9a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:23.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 shutdown_connections 2023-11-15T05:07:23.529 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:23.526+0000 7ff9aa54f700 1 -- 172.21.15.154:0/945446876 wait complete. 2023-11-15T05:07:23.529 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:24.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:23.526297+0000 mon.smithi086 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.154:0/945446876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:24.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:24.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:24.853257+0000 mgr.smithi086.hkfamo (mgr.14162) 185 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:26.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 -- 172.21.15.154:0/1544240664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 msgr2=0x7f79400f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:26.054 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 --2- 172.21.15.154:0/1544240664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79400f9150 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f7934009fb0 tx=0x7f7934009b30).stop 2023-11-15T05:07:26.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 -- 172.21.15.154:0/1544240664 shutdown_connections 2023-11-15T05:07:26.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 --2- 172.21.15.154:0/1544240664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79400f9150 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:26.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 -- 172.21.15.154:0/1544240664 >> 172.21.15.154:0/1544240664 conn(0x7f79400f0b70 msgr2=0x7f79400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:26.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 -- 172.21.15.154:0/1544240664 shutdown_connections 2023-11-15T05:07:26.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 -- 172.21.15.154:0/1544240664 wait complete. 2023-11-15T05:07:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.047+0000 7f7945e6c700 1 Processor -- start 2023-11-15T05:07:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f7945e6c700 1 -- start start 2023-11-15T05:07:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f7945e6c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f7945e6c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940106bd0 con 0x7f7940060220 2023-11-15T05:07:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f793f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:26.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f793f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40914/0 (socket says 172.21.15.154:40914) 2023-11-15T05:07:26.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f793f7fe700 1 -- 172.21.15.154:0/2429107801 learned_addr learned my addr 172.21.15.154:0/2429107801 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:26.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f793f7fe700 1 -- 172.21.15.154:0/2429107801 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934005040 con 0x7f7940060220 2023-11-15T05:07:26.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.048+0000 7f793f7fe700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7934009830 tx=0x7f793400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:26.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7934009320 con 0x7f7940060220 2023-11-15T05:07:26.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940106dd0 con 0x7f7940060220 2023-11-15T05:07:26.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79340136c0 con 0x7f7940060220 2023-11-15T05:07:26.061 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793401ee90 con 0x7f7940060220 2023-11-15T05:07:26.061 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7940107260 con 0x7f7940060220 2023-11-15T05:07:26.061 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.049+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f793401e420 con 0x7f7940060220 2023-11-15T05:07:26.062 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.050+0000 7f7927fff700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 0x7f79280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:26.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.050+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940045ad0 con 0x7f7940060220 2023-11-15T05:07:26.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.050+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79340410b0 con 0x7f7940060220 2023-11-15T05:07:26.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.050+0000 7f793effd700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 0x7f79280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:26.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.050+0000 7f793effd700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 0x7f79280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f79300096a0 tx=0x7f7930006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:26.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.055+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f793401cb10 con 0x7f7940060220 2023-11-15T05:07:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:24.931405+0000 mon.smithi086 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:07:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:24.937819+0000 mon.smithi086 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:07:26.264 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.261+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7940108110 con 0x7f7940060220 2023-11-15T05:07:26.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.262+0000 7f7927fff700 1 -- 172.21.15.154:0/2429107801 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7934024d40 con 0x7f7940060220 2023-11-15T05:07:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:26.266 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:26.267 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 msgr2=0x7f79280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:26.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 0x7f79280323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f79300096a0 tx=0x7f7930006b40).stop 2023-11-15T05:07:26.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 msgr2=0x7f79401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:26.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7934009830 tx=0x7f793400b040).stop 2023-11-15T05:07:26.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 shutdown_connections 2023-11-15T05:07:26.268 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f792802ff10 0x7f79280323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:26.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 --2- 172.21.15.154:0/2429107801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7940060220 0x7f79401066e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:26.271 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 >> 172.21.15.154:0/2429107801 conn(0x7f79400f0b70 msgr2=0x7f79400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:26.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 shutdown_connections 2023-11-15T05:07:26.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:26.264+0000 7f7945e6c700 1 -- 172.21.15.154:0/2429107801 wait complete. 2023-11-15T05:07:26.272 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:27.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:26 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:26.264232+0000 mon.smithi086 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.154:0/2429107801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:27.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:26 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:26.853464+0000 mgr.smithi086.hkfamo (mgr.14162) 186 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:27.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:27.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:28.794 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 -- 172.21.15.154:0/1218305034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 msgr2=0x7f72fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:28.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 --2- 172.21.15.154:0/1218305034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc0f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f72e4009fb0 tx=0x7f72e4009b30).stop 2023-11-15T05:07:28.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 -- 172.21.15.154:0/1218305034 shutdown_connections 2023-11-15T05:07:28.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 --2- 172.21.15.154:0/1218305034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc0f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:28.795 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 -- 172.21.15.154:0/1218305034 >> 172.21.15.154:0/1218305034 conn(0x7f72fc0f0b70 msgr2=0x7f72fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:28.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 -- 172.21.15.154:0/1218305034 shutdown_connections 2023-11-15T05:07:28.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.792+0000 7f7300d3e700 1 -- 172.21.15.154:0/1218305034 wait complete. 2023-11-15T05:07:28.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f7300d3e700 1 Processor -- start 2023-11-15T05:07:28.796 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f7300d3e700 1 -- start start 2023-11-15T05:07:28.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f7300d3e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:28.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f7300d3e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc106c20 con 0x7f72fc0f5410 2023-11-15T05:07:28.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f72fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:28.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f72fa59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40930/0 (socket says 172.21.15.154:40930) 2023-11-15T05:07:28.797 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.793+0000 7f72fa59c700 1 -- 172.21.15.154:0/4277348461 learned_addr learned my addr 172.21.15.154:0/4277348461 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f72fa59c700 1 -- 172.21.15.154:0/4277348461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e4005040 con 0x7f72fc0f5410 2023-11-15T05:07:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f72fa59c700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f72e4014000 tx=0x7f72e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e40095f0 con 0x7f72fc0f5410 2023-11-15T05:07:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72e4007e70 con 0x7f72fc0f5410 2023-11-15T05:07:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc106e20 con 0x7f72fc0f5410 2023-11-15T05:07:28.799 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e4014ea0 con 0x7f72fc0f5410 2023-11-15T05:07:28.799 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.794+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1072b0 con 0x7f72fc0f5410 2023-11-15T05:07:28.800 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72e4025070 con 0x7f72fc0f5410 2023-11-15T05:07:28.800 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f72f2ffd700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 0x7f72e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:28.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc045ad0 con 0x7f72fc0f5410 2023-11-15T05:07:28.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72e40409e0 con 0x7f72fc0f5410 2023-11-15T05:07:28.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f72f9d9b700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 0x7f72e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:28.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.795+0000 7f72f9d9b700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 0x7f72e8032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f72ec0096a0 tx=0x7f72ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:28.801 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:28.800+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72e400f300 con 0x7f72fc0f5410 2023-11-15T05:07:29.006 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.005+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72fc0faf50 con 0x7f72fc0f5410 2023-11-15T05:07:29.007 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.006+0000 7f72f2ffd700 1 -- 172.21.15.154:0/4277348461 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72e400f300 con 0x7f72fc0f5410 2023-11-15T05:07:29.007 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:29.008 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:29.009 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 msgr2=0x7f72e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:29.009 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 0x7f72e8032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f72ec0096a0 tx=0x7f72ec006b40).stop 2023-11-15T05:07:29.009 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 msgr2=0x7f72fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:29.009 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f72e4014000 tx=0x7f72e400b040).stop 2023-11-15T05:07:29.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 shutdown_connections 2023-11-15T05:07:29.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f72e802ffb0 0x7f72e8032470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:29.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 --2- 172.21.15.154:0/4277348461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72fc0f5410 0x7f72fc106730 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:29.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 >> 172.21.15.154:0/4277348461 conn(0x7f72fc0f0b70 msgr2=0x7f72fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:29.010 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 shutdown_connections 2023-11-15T05:07:29.011 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:29.008+0000 7f7300d3e700 1 -- 172.21.15.154:0/4277348461 wait complete. 2023-11-15T05:07:29.011 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:29.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:28.853681+0000 mgr.smithi086.hkfamo (mgr.14162) 187 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:30.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:29.006857+0000 mon.smithi086 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.154:0/4277348461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:30.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:30.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:30.853901+0000 mgr.smithi086.hkfamo (mgr.14162) 188 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:31.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 -- 172.21.15.154:0/3520441173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 msgr2=0x7f40000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:31.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 --2- 172.21.15.154:0/3520441173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f40000f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3fe8009fb0 tx=0x7f3fe8009b30).stop 2023-11-15T05:07:31.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 -- 172.21.15.154:0/3520441173 shutdown_connections 2023-11-15T05:07:31.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 --2- 172.21.15.154:0/3520441173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f40000f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:31.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 -- 172.21.15.154:0/3520441173 >> 172.21.15.154:0/3520441173 conn(0x7f40000f1c70 msgr2=0x7f40000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:31.493 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 -- 172.21.15.154:0/3520441173 shutdown_connections 2023-11-15T05:07:31.493 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.488+0000 7f4006030700 1 -- 172.21.15.154:0/3520441173 wait complete. 2023-11-15T05:07:31.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.489+0000 7f4006030700 1 Processor -- start 2023-11-15T05:07:31.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.489+0000 7f4006030700 1 -- start start 2023-11-15T05:07:31.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.489+0000 7f4006030700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:31.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.489+0000 7f4006030700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4000106c20 con 0x7f40000f6510 2023-11-15T05:07:31.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3fff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:31.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3fff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:40942/0 (socket says 172.21.15.154:40942) 2023-11-15T05:07:31.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3fff7fe700 1 -- 172.21.15.154:0/461389437 learned_addr learned my addr 172.21.15.154:0/461389437 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:31.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3fff7fe700 1 -- 172.21.15.154:0/461389437 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8005040 con 0x7f40000f6510 2023-11-15T05:07:31.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3fff7fe700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3fe8013510 tx=0x7f3fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:31.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8009320 con 0x7f40000f6510 2023-11-15T05:07:31.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fe8013f00 con 0x7f40000f6510 2023-11-15T05:07:31.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4000106e20 con 0x7f40000f6510 2023-11-15T05:07:31.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8014d90 con 0x7f40000f6510 2023-11-15T05:07:31.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.490+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40001072b0 con 0x7f40000f6510 2023-11-15T05:07:31.498 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.491+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fe80263a0 con 0x7f40000f6510 2023-11-15T05:07:31.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.491+0000 7f3ff7fff700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:31.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.491+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fe8041020 con 0x7f40000f6510 2023-11-15T05:07:31.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.491+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000045ad0 con 0x7f40000f6510 2023-11-15T05:07:31.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.491+0000 7f3ffeffd700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:31.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.492+0000 7f3ffeffd700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 0x7f3fec032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3ff00096a0 tx=0x7f3ff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:31.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.496+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fe8012920 con 0x7f40000f6510 2023-11-15T05:07:31.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.703+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40000fc050 con 0x7f40000f6510 2023-11-15T05:07:31.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.704+0000 7f3ff7fff700 1 -- 172.21.15.154:0/461389437 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fe802b020 con 0x7f40000f6510 2023-11-15T05:07:31.705 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:31.705 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:31.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 msgr2=0x7f3fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:31.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 0x7f3fec032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3ff00096a0 tx=0x7f3ff0006b40).stop 2023-11-15T05:07:31.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 msgr2=0x7f4000106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:31.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3fe8013510 tx=0x7f3fe800b040).stop 2023-11-15T05:07:31.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 shutdown_connections 2023-11-15T05:07:31.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3fec02ff60 0x7f3fec032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:31.707 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 --2- 172.21.15.154:0/461389437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000f6510 0x7f4000106730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:31.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.705+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 >> 172.21.15.154:0/461389437 conn(0x7f40000f1c70 msgr2=0x7f40000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:31.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.706+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 shutdown_connections 2023-11-15T05:07:31.708 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:31.706+0000 7f4006030700 1 -- 172.21.15.154:0/461389437 wait complete. 2023-11-15T05:07:31.709 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:32.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:31.704922+0000 mon.smithi086 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.154:0/461389437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:33.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:33.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:33.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:32.854118+0000 mgr.smithi086.hkfamo (mgr.14162) 189 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:34.263 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 -- 172.21.15.154:0/2758639154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 msgr2=0x7fca380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:34.264 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/2758639154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca380f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fca2c009fb0 tx=0x7fca2c009b30).stop 2023-11-15T05:07:34.264 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 -- 172.21.15.154:0/2758639154 shutdown_connections 2023-11-15T05:07:34.264 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/2758639154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca380f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:34.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 -- 172.21.15.154:0/2758639154 >> 172.21.15.154:0/2758639154 conn(0x7fca380f1c70 msgr2=0x7fca380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:34.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 -- 172.21.15.154:0/2758639154 shutdown_connections 2023-11-15T05:07:34.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.261+0000 7fca3ccdf700 1 -- 172.21.15.154:0/2758639154 wait complete. 2023-11-15T05:07:34.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3ccdf700 1 Processor -- start 2023-11-15T05:07:34.265 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3ccdf700 1 -- start start 2023-11-15T05:07:34.266 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3ccdf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:34.266 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3ccdf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38106c20 con 0x7fca380f6510 2023-11-15T05:07:34.266 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:34.266 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54938/0 (socket says 172.21.15.154:54938) 2023-11-15T05:07:34.267 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.262+0000 7fca3659c700 1 -- 172.21.15.154:0/1097767702 learned_addr learned my addr 172.21.15.154:0/1097767702 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:34.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca3659c700 1 -- 172.21.15.154:0/1097767702 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c005040 con 0x7fca380f6510 2023-11-15T05:07:34.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca3659c700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fca2c014000 tx=0x7fca2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:34.269 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca2c0095f0 con 0x7fca380f6510 2023-11-15T05:07:34.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca38106e20 con 0x7fca380f6510 2023-11-15T05:07:34.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca2c007e70 con 0x7fca380f6510 2023-11-15T05:07:34.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca2c014ea0 con 0x7fca380f6510 2023-11-15T05:07:34.270 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.263+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca381072b0 con 0x7fca380f6510 2023-11-15T05:07:34.271 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.264+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca2c025070 con 0x7fca380f6510 2023-11-15T05:07:34.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.264+0000 7fca26ffd700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:34.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.264+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca2c040a40 con 0x7fca380f6510 2023-11-15T05:07:34.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.264+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca38045ad0 con 0x7fca380f6510 2023-11-15T05:07:34.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.264+0000 7fca35d9b700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:34.273 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.265+0000 7fca35d9b700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 0x7fca20032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:34.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.272+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca2c026350 con 0x7fca380f6510 2023-11-15T05:07:34.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.480+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca38107ff0 con 0x7fca380f6510 2023-11-15T05:07:34.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.480+0000 7fca26ffd700 1 -- 172.21.15.154:0/1097767702 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca2c025300 con 0x7fca380f6510 2023-11-15T05:07:34.482 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:34.482 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:34.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 msgr2=0x7fca20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:34.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 0x7fca20032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).stop 2023-11-15T05:07:34.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 msgr2=0x7fca38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fca2c014000 tx=0x7fca2c00b040).stop 2023-11-15T05:07:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 shutdown_connections 2023-11-15T05:07:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fca2002ff60 0x7fca20032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 --2- 172.21.15.154:0/1097767702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca380f6510 0x7fca38106730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 >> 172.21.15.154:0/1097767702 conn(0x7fca380f1c70 msgr2=0x7fca380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:34.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 shutdown_connections 2023-11-15T05:07:34.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:34.482+0000 7fca3ccdf700 1 -- 172.21.15.154:0/1097767702 wait complete. 2023-11-15T05:07:34.485 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:34.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:34.481479+0000 mon.smithi086 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.154:0/1097767702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:35.659 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:34.854320+0000 mgr.smithi086.hkfamo (mgr.14162) 190 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:35.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:35.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:36.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 -- 172.21.15.154:0/1548631701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 msgr2=0x7f1ba40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:36.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 --2- 172.21.15.154:0/1548631701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba40f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f1b94009fb0 tx=0x7f1b94009b30).stop 2023-11-15T05:07:36.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 -- 172.21.15.154:0/1548631701 shutdown_connections 2023-11-15T05:07:36.985 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 --2- 172.21.15.154:0/1548631701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba40f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 -- 172.21.15.154:0/1548631701 >> 172.21.15.154:0/1548631701 conn(0x7f1ba40f0b70 msgr2=0x7f1ba40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:36.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 -- 172.21.15.154:0/1548631701 shutdown_connections 2023-11-15T05:07:36.990 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.982+0000 7f1babd53700 1 -- 172.21.15.154:0/1548631701 wait complete. 2023-11-15T05:07:36.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1babd53700 1 Processor -- start 2023-11-15T05:07:36.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1babd53700 1 -- start start 2023-11-15T05:07:36.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1babd53700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:36.991 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1babd53700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4106c10 con 0x7f1ba40f5410 2023-11-15T05:07:36.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1ba9aef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:36.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1ba9aef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54952/0 (socket says 172.21.15.154:54952) 2023-11-15T05:07:36.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.983+0000 7f1ba9aef700 1 -- 172.21.15.154:0/1374265064 learned_addr learned my addr 172.21.15.154:0/1374265064 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:36.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1ba9aef700 1 -- 172.21.15.154:0/1374265064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b94005040 con 0x7f1ba40f5410 2023-11-15T05:07:36.992 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1ba9aef700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1b94014000 tx=0x7f1b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:36.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b940095f0 con 0x7f1ba40f5410 2023-11-15T05:07:36.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba4106e10 con 0x7f1ba40f5410 2023-11-15T05:07:36.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b94007e70 con 0x7f1ba40f5410 2023-11-15T05:07:36.993 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b94014ea0 con 0x7f1ba40f5410 2023-11-15T05:07:36.994 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.984+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba41072a0 con 0x7f1ba40f5410 2023-11-15T05:07:36.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b94025070 con 0x7f1ba40f5410 2023-11-15T05:07:36.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba4045ad0 con 0x7f1ba40f5410 2023-11-15T05:07:36.995 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1b9a7fc700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 0x7f1b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:36.996 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b94040d40 con 0x7f1ba40f5410 2023-11-15T05:07:36.996 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1ba92ee700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 0x7f1b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:36.997 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.985+0000 7f1ba92ee700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 0x7f1b90032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1ba00096a0 tx=0x7f1ba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:36.997 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:36.990+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b94025950 con 0x7f1ba40f5410 2023-11-15T05:07:37.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.197+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ba4108150 con 0x7f1ba40f5410 2023-11-15T05:07:37.200 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.197+0000 7f1b9a7fc700 1 -- 172.21.15.154:0/1374265064 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b94025950 con 0x7f1ba40f5410 2023-11-15T05:07:37.202 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:37.202 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:37.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 msgr2=0x7f1b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:37.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 0x7f1b90032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1ba00096a0 tx=0x7f1ba0006b40).stop 2023-11-15T05:07:37.204 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 msgr2=0x7f1ba4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:37.204 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1b94014000 tx=0x7f1b9400b040).stop 2023-11-15T05:07:37.204 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 shutdown_connections 2023-11-15T05:07:37.204 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1b9002ff60 0x7f1b90032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:37.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 --2- 172.21.15.154:0/1374265064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ba40f5410 0x7f1ba4106720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:37.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 >> 172.21.15.154:0/1374265064 conn(0x7f1ba40f0b70 msgr2=0x7f1ba405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:37.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 shutdown_connections 2023-11-15T05:07:37.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:37.199+0000 7f1babd53700 1 -- 172.21.15.154:0/1374265064 wait complete. 2023-11-15T05:07:37.206 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:38.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:36.758097+0000 mon.smithi086 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' 2023-11-15T05:07:38.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:07:36.759448+0000 mgr.smithi086.hkfamo (mgr.14162) 191 : cephadm [INF] Deploying daemon node-exporter.smithi086 on smithi086 2023-11-15T05:07:38.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:36.854546+0000 mgr.smithi086.hkfamo (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:38.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:37.198390+0000 mon.smithi086 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.154:0/1374265064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:38.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:38.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:39.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:38.854826+0000 mgr.smithi086.hkfamo (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:39.735 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 -- 172.21.15.154:0/983057036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 msgr2=0x7f5b080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/983057036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b080f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5afc009fb0 tx=0x7f5afc009b30).stop 2023-11-15T05:07:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 -- 172.21.15.154:0/983057036 shutdown_connections 2023-11-15T05:07:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/983057036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b080f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 -- 172.21.15.154:0/983057036 >> 172.21.15.154:0/983057036 conn(0x7f5b080f1c70 msgr2=0x7f5b080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 -- 172.21.15.154:0/983057036 shutdown_connections 2023-11-15T05:07:39.737 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.734+0000 7f5b0de2f700 1 -- 172.21.15.154:0/983057036 wait complete. 2023-11-15T05:07:39.737 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b0de2f700 1 Processor -- start 2023-11-15T05:07:39.737 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b0de2f700 1 -- start start 2023-11-15T05:07:39.737 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b0de2f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:39.740 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b0de2f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08106c20 con 0x7f5b080f6510 2023-11-15T05:07:39.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b077fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:39.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b077fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54974/0 (socket says 172.21.15.154:54974) 2023-11-15T05:07:39.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.735+0000 7f5b077fe700 1 -- 172.21.15.154:0/4008967908 learned_addr learned my addr 172.21.15.154:0/4008967908 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:39.741 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5b077fe700 1 -- 172.21.15.154:0/4008967908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc005040 con 0x7f5b080f6510 2023-11-15T05:07:39.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5b077fe700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f5afc014000 tx=0x7f5afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:39.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc0095f0 con 0x7f5b080f6510 2023-11-15T05:07:39.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b08106e20 con 0x7f5b080f6510 2023-11-15T05:07:39.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5afc007e70 con 0x7f5b080f6510 2023-11-15T05:07:39.742 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc014ea0 con 0x7f5b080f6510 2023-11-15T05:07:39.743 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.736+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b081072b0 con 0x7f5b080f6510 2023-11-15T05:07:39.743 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.737+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5afc025070 con 0x7f5b080f6510 2023-11-15T05:07:39.743 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.737+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08045ad0 con 0x7f5b080f6510 2023-11-15T05:07:39.743 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.737+0000 7f5aeffff700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 0x7f5af0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:39.743 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.737+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5afc040930 con 0x7f5b080f6510 2023-11-15T05:07:39.744 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.737+0000 7f5b06ffd700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 0x7f5af0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:39.744 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.738+0000 7f5b06ffd700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 0x7f5af0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5af80096a0 tx=0x7f5af8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:39.744 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.741+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5afc00f4e0 con 0x7f5b080f6510 2023-11-15T05:07:39.948 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.948+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b080fbf60 con 0x7f5b080f6510 2023-11-15T05:07:39.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.948+0000 7f5aeffff700 1 -- 172.21.15.154:0/4008967908 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5afc00f300 con 0x7f5b080f6510 2023-11-15T05:07:39.949 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:39.950 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:39.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 msgr2=0x7f5af0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:39.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 0x7f5af0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5af80096a0 tx=0x7f5af8006b40).stop 2023-11-15T05:07:39.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 msgr2=0x7f5b08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:39.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f5afc014000 tx=0x7f5afc00b040).stop 2023-11-15T05:07:39.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 shutdown_connections 2023-11-15T05:07:39.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5af002ff60 0x7f5af0032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:39.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 --2- 172.21.15.154:0/4008967908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b080f6510 0x7f5b08106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:39.953 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 >> 172.21.15.154:0/4008967908 conn(0x7f5b080f1c70 msgr2=0x7f5b080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:39.953 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 shutdown_connections 2023-11-15T05:07:39.953 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:39.950+0000 7f5b0de2f700 1 -- 172.21.15.154:0/4008967908 wait complete. 2023-11-15T05:07:39.953 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:40.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:39.949481+0000 mon.smithi086 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.154:0/4008967908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:41.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:41.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:41.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:41 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:40.855130+0000 mgr.smithi086.hkfamo (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:42.569 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 -- 172.21.15.154:0/1502929400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 msgr2=0x7f6114069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:42.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 --2- 172.21.15.154:0/1502929400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f6114069e60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6104009fb0 tx=0x7f6104009b30).stop 2023-11-15T05:07:42.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 -- 172.21.15.154:0/1502929400 shutdown_connections 2023-11-15T05:07:42.570 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 --2- 172.21.15.154:0/1502929400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f6114069e60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:42.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 -- 172.21.15.154:0/1502929400 >> 172.21.15.154:0/1502929400 conn(0x7f61140f1c70 msgr2=0x7f61140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:42.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 -- 172.21.15.154:0/1502929400 shutdown_connections 2023-11-15T05:07:42.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.568+0000 7f6118b38700 1 -- 172.21.15.154:0/1502929400 wait complete. 2023-11-15T05:07:42.571 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f6118b38700 1 Processor -- start 2023-11-15T05:07:42.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f6118b38700 1 -- start start 2023-11-15T05:07:42.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f6118b38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:42.572 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f6118b38700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6114108db0 con 0x7f611406b960 2023-11-15T05:07:42.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f611259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:42.574 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f611259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54988/0 (socket says 172.21.15.154:54988) 2023-11-15T05:07:42.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.569+0000 7f611259c700 1 -- 172.21.15.154:0/4176669854 learned_addr learned my addr 172.21.15.154:0/4176669854 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:42.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f611259c700 1 -- 172.21.15.154:0/4176669854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6104005040 con 0x7f611406b960 2023-11-15T05:07:42.575 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f611259c700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f6104013510 tx=0x7f610400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:42.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6104009320 con 0x7f611406b960 2023-11-15T05:07:42.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6104013f00 con 0x7f611406b960 2023-11-15T05:07:42.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6114108fb0 con 0x7f611406b960 2023-11-15T05:07:42.576 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6104014dc0 con 0x7f611406b960 2023-11-15T05:07:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.570+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6114109440 con 0x7f611406b960 2023-11-15T05:07:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.571+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f610401d3d0 con 0x7f611406b960 2023-11-15T05:07:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.571+0000 7f6102ffd700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.571+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6104041100 con 0x7f611406b960 2023-11-15T05:07:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.571+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6114045ad0 con 0x7f611406b960 2023-11-15T05:07:42.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.571+0000 7f6111d9b700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:42.578 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.572+0000 7f6111d9b700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 0x7f60fc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f61080096a0 tx=0x7f6108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:42.579 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.576+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6104012920 con 0x7f611406b960 2023-11-15T05:07:42.780 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.780+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61140fe240 con 0x7f611406b960 2023-11-15T05:07:42.781 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.781+0000 7f6102ffd700 1 -- 172.21.15.154:0/4176669854 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f610402b020 con 0x7f611406b960 2023-11-15T05:07:42.782 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:42.782 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:42.783 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 msgr2=0x7f60fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:42.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 0x7f60fc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f61080096a0 tx=0x7f6108006b40).stop 2023-11-15T05:07:42.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 msgr2=0x7f61141088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:42.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f6104013510 tx=0x7f610400b040).stop 2023-11-15T05:07:42.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 shutdown_connections 2023-11-15T05:07:42.784 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:42.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 --2- 172.21.15.154:0/4176669854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f611406b960 0x7f61141088c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:42.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 >> 172.21.15.154:0/4176669854 conn(0x7f61140f1c70 msgr2=0x7f61140f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:42.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 shutdown_connections 2023-11-15T05:07:42.785 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:42.783+0000 7f6118b38700 1 -- 172.21.15.154:0/4176669854 wait complete. 2023-11-15T05:07:42.786 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:43.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:42.781729+0000 mon.smithi086 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.154:0/4176669854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:44.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:44.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:44.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:42.855411+0000 mgr.smithi086.hkfamo (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:44.855730+0000 mgr.smithi086.hkfamo (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:45.311 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 -- 172.21.15.154:0/1196292142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 msgr2=0x7ff1780f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:45.311 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/1196292142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1780f90f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff160009fb0 tx=0x7ff160009b30).stop 2023-11-15T05:07:45.311 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 -- 172.21.15.154:0/1196292142 shutdown_connections 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/1196292142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1780f90f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 -- 172.21.15.154:0/1196292142 >> 172.21.15.154:0/1196292142 conn(0x7ff1780f0b70 msgr2=0x7ff1780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 -- 172.21.15.154:0/1196292142 shutdown_connections 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 -- 172.21.15.154:0/1196292142 wait complete. 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.309+0000 7ff17dd7a700 1 Processor -- start 2023-11-15T05:07:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff17dd7a700 1 -- start start 2023-11-15T05:07:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff17dd7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff17dd7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff178106bd0 con 0x7ff178060280 2023-11-15T05:07:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff1777fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff1777fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54238/0 (socket says 172.21.15.154:54238) 2023-11-15T05:07:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff1777fe700 1 -- 172.21.15.154:0/4084204224 learned_addr learned my addr 172.21.15.154:0/4084204224 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:45.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.310+0000 7ff1777fe700 1 -- 172.21.15.154:0/4084204224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff160005040 con 0x7ff178060280 2023-11-15T05:07:45.314 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff1777fe700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff160009830 tx=0x7ff16000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:45.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff160009320 con 0x7ff178060280 2023-11-15T05:07:45.315 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff178106dd0 con 0x7ff178060280 2023-11-15T05:07:45.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff178107260 con 0x7ff178060280 2023-11-15T05:07:45.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff160004490 con 0x7ff178060280 2023-11-15T05:07:45.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.311+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff16001ee20 con 0x7ff178060280 2023-11-15T05:07:45.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff16001e420 con 0x7ff178060280 2023-11-15T05:07:45.316 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c004fa0 con 0x7ff178060280 2023-11-15T05:07:45.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff16ffff700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 0x7ff164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:45.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff160041050 con 0x7ff178060280 2023-11-15T05:07:45.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff176ffd700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 0x7ff164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:45.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.312+0000 7ff176ffd700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 0x7ff164032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff1680096a0 tx=0x7ff168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:45.317 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.316+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff16001c790 con 0x7ff178060280 2023-11-15T05:07:45.522 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.522+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff15c005670 con 0x7ff178060280 2023-11-15T05:07:45.523 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.522+0000 7ff16ffff700 1 -- 172.21.15.154:0/4084204224 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff16002b2d0 con 0x7ff178060280 2023-11-15T05:07:45.523 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:45.524 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:45.525 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 msgr2=0x7ff164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:45.525 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 0x7ff164032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff1680096a0 tx=0x7ff168006b40).stop 2023-11-15T05:07:45.525 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 msgr2=0x7ff1781066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:45.525 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7ff160009830 tx=0x7ff16000b040).stop 2023-11-15T05:07:45.525 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 shutdown_connections 2023-11-15T05:07:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff16402ff60 0x7ff164032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 --2- 172.21.15.154:0/4084204224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff178060280 0x7ff1781066e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 >> 172.21.15.154:0/4084204224 conn(0x7ff1780f0b70 msgr2=0x7ff1780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 shutdown_connections 2023-11-15T05:07:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:45.524+0000 7ff17dd7a700 1 -- 172.21.15.154:0/4084204224 wait complete. 2023-11-15T05:07:45.527 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:46.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:45.523584+0000 mon.smithi086 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.154:0/4084204224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:46.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:46.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:46.856143+0000 mgr.smithi086.hkfamo (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:48.103 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 -- 172.21.15.154:0/551993816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 msgr2=0x7f49f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 --2- 172.21.15.154:0/551993816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f80f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f49e8009fb0 tx=0x7f49e8009b30).stop 2023-11-15T05:07:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 -- 172.21.15.154:0/551993816 shutdown_connections 2023-11-15T05:07:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 --2- 172.21.15.154:0/551993816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f80f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:48.104 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 -- 172.21.15.154:0/551993816 >> 172.21.15.154:0/551993816 conn(0x7f49f80f1c70 msgr2=0x7f49f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 -- 172.21.15.154:0/551993816 shutdown_connections 2023-11-15T05:07:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.102+0000 7f49fe375700 1 -- 172.21.15.154:0/551993816 wait complete. 2023-11-15T05:07:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.103+0000 7f49fe375700 1 Processor -- start 2023-11-15T05:07:48.105 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.103+0000 7f49fe375700 1 -- start start 2023-11-15T05:07:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.103+0000 7f49fe375700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.103+0000 7f49fe375700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8069e20 con 0x7f49f80f6510 2023-11-15T05:07:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:48.107 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54260/0 (socket says 172.21.15.154:54260) 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f7fff700 1 -- 172.21.15.154:0/3206079637 learned_addr learned my addr 172.21.15.154:0/3206079637 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f7fff700 1 -- 172.21.15.154:0/3206079637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e8005040 con 0x7f49f80f6510 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f7fff700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f49e8013510 tx=0x7f49e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e8009320 con 0x7f49f80f6510 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49e8013f00 con 0x7f49f80f6510 2023-11-15T05:07:48.108 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f8069fc0 con 0x7f49f80f6510 2023-11-15T05:07:48.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e8014d90 con 0x7f49f80f6510 2023-11-15T05:07:48.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.104+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f806a450 con 0x7f49f80f6510 2023-11-15T05:07:48.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.105+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49e8014420 con 0x7f49f80f6510 2023-11-15T05:07:48.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.105+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f8059100 con 0x7f49f80f6510 2023-11-15T05:07:48.109 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.105+0000 7f49f4ff9700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 0x7f49e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:48.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.105+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49e8040fe0 con 0x7f49f80f6510 2023-11-15T05:07:48.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.105+0000 7f49f77fe700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 0x7f49e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:48.110 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.106+0000 7f49f77fe700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 0x7f49e0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f49ec0096a0 tx=0x7f49ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:48.111 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.110+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49e80095f0 con 0x7f49f80f6510 2023-11-15T05:07:48.318 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.317+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49f80fbf60 con 0x7f49f80f6510 2023-11-15T05:07:48.319 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.318+0000 7f49f4ff9700 1 -- 172.21.15.154:0/3206079637 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49e801db70 con 0x7f49f80f6510 2023-11-15T05:07:48.319 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:48.320 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:48.320 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 msgr2=0x7f49e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:48.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 0x7f49e0032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f49ec0096a0 tx=0x7f49ec006b40).stop 2023-11-15T05:07:48.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 msgr2=0x7f49f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:48.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f49e8013510 tx=0x7f49e800b040).stop 2023-11-15T05:07:48.321 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 shutdown_connections 2023-11-15T05:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f49e002ff60 0x7f49e0032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 --2- 172.21.15.154:0/3206079637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49f80f6510 0x7f49f806b9a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 >> 172.21.15.154:0/3206079637 conn(0x7f49f80f1c70 msgr2=0x7f49f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 shutdown_connections 2023-11-15T05:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:48.320+0000 7f49fe375700 1 -- 172.21.15.154:0/3206079637 wait complete. 2023-11-15T05:07:48.323 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:48.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:48.319288+0000 mon.smithi086 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.154:0/3206079637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:49.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:49.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:49.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:48.856530+0000 mgr.smithi086.hkfamo (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 -- 172.21.15.154:0/613817055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 msgr2=0x7ffba00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/613817055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba00f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ffb94009fb0 tx=0x7ffb94009b30).stop 2023-11-15T05:07:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 -- 172.21.15.154:0/613817055 shutdown_connections 2023-11-15T05:07:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/613817055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba00f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 -- 172.21.15.154:0/613817055 >> 172.21.15.154:0/613817055 conn(0x7ffba00f0b70 msgr2=0x7ffba00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 -- 172.21.15.154:0/613817055 shutdown_connections 2023-11-15T05:07:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 -- 172.21.15.154:0/613817055 wait complete. 2023-11-15T05:07:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.918+0000 7ffba6bb3700 1 Processor -- start 2023-11-15T05:07:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba6bb3700 1 -- start start 2023-11-15T05:07:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba6bb3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba6bb3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba0060220 con 0x7ffba00f5410 2023-11-15T05:07:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba494f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba494f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:54290/0 (socket says 172.21.15.154:54290) 2023-11-15T05:07:50.923 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba494f700 1 -- 172.21.15.154:0/1839268441 learned_addr learned my addr 172.21.15.154:0/1839268441 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:50.923 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba494f700 1 -- 172.21.15.154:0/1839268441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94005040 con 0x7ffba00f5410 2023-11-15T05:07:50.923 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.919+0000 7ffba494f700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ffb94013510 tx=0x7ffb9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:50.924 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.921+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb94009320 con 0x7ffba00f5410 2023-11-15T05:07:50.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.921+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb94013f00 con 0x7ffba00f5410 2023-11-15T05:07:50.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.921+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba0060420 con 0x7ffba00f5410 2023-11-15T05:07:50.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.921+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb94014d90 con 0x7ffba00f5410 2023-11-15T05:07:50.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.921+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba0109b00 con 0x7ffba00f5410 2023-11-15T05:07:50.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.922+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb940263a0 con 0x7ffba00f5410 2023-11-15T05:07:50.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.922+0000 7ffb9d7fa700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:50.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.922+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb940410d0 con 0x7ffba00f5410 2023-11-15T05:07:50.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.922+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba0059100 con 0x7ffba00f5410 2023-11-15T05:07:50.926 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.922+0000 7ffb9ffff700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:50.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.923+0000 7ffb9ffff700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 0x7ffb90032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffb8c0096a0 tx=0x7ffb8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:50.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:50.927+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb94012920 con 0x7ffba00f5410 2023-11-15T05:07:51.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.130+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffba010a1e0 con 0x7ffba00f5410 2023-11-15T05:07:51.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.131+0000 7ffb9d7fa700 1 -- 172.21.15.154:0/1839268441 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb9402b020 con 0x7ffba00f5410 2023-11-15T05:07:51.132 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:51.132 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:51.133 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.132+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 msgr2=0x7ffb90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:51.133 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.132+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 0x7ffb90032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffb8c0096a0 tx=0x7ffb8c006b40).stop 2023-11-15T05:07:51.133 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.132+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 msgr2=0x7ffba0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.132+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ffb94013510 tx=0x7ffb9400b040).stop 2023-11-15T05:07:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 shutdown_connections 2023-11-15T05:07:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ffb9002ff60 0x7ffb90032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 --2- 172.21.15.154:0/1839268441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffba00f5410 0x7ffba0109450 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 >> 172.21.15.154:0/1839268441 conn(0x7ffba00f0b70 msgr2=0x7ffba00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:51.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 shutdown_connections 2023-11-15T05:07:51.135 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:51.133+0000 7ffba6bb3700 1 -- 172.21.15.154:0/1839268441 wait complete. 2023-11-15T05:07:51.135 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:51.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:50.856885+0000 mgr.smithi086.hkfamo (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:52.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:51.131945+0000 mon.smithi086 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.154:0/1839268441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:52.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:52.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:53.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:52.857281+0000 mgr.smithi086.hkfamo (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:53.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 -- 172.21.15.154:0/2177503548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 msgr2=0x7f34340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:53.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 --2- 172.21.15.154:0/2177503548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f34340f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3428009fb0 tx=0x7f3428009b30).stop 2023-11-15T05:07:53.624 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 -- 172.21.15.154:0/2177503548 shutdown_connections 2023-11-15T05:07:53.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 --2- 172.21.15.154:0/2177503548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f34340f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:53.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 -- 172.21.15.154:0/2177503548 >> 172.21.15.154:0/2177503548 conn(0x7f34340f1c70 msgr2=0x7f34340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:53.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 -- 172.21.15.154:0/2177503548 shutdown_connections 2023-11-15T05:07:53.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 -- 172.21.15.154:0/2177503548 wait complete. 2023-11-15T05:07:53.625 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.622+0000 7f343a876700 1 Processor -- start 2023-11-15T05:07:53.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f343a876700 1 -- start start 2023-11-15T05:07:53.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f343a876700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:53.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f343a876700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3434106c20 con 0x7f34340f6510 2023-11-15T05:07:53.626 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3433fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:53.627 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3433fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53852/0 (socket says 172.21.15.154:53852) 2023-11-15T05:07:53.627 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3433fff700 1 -- 172.21.15.154:0/883291767 learned_addr learned my addr 172.21.15.154:0/883291767 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:53.627 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3433fff700 1 -- 172.21.15.154:0/883291767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428005040 con 0x7f34340f6510 2023-11-15T05:07:53.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3433fff700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3428013510 tx=0x7f342800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:53.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428009320 con 0x7f34340f6510 2023-11-15T05:07:53.628 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3428013f00 con 0x7f34340f6510 2023-11-15T05:07:53.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.623+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3434106e20 con 0x7f34340f6510 2023-11-15T05:07:53.629 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.624+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428014d90 con 0x7f34340f6510 2023-11-15T05:07:53.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.624+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34341072b0 con 0x7f34340f6510 2023-11-15T05:07:53.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.624+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34280095f0 con 0x7f34340f6510 2023-11-15T05:07:53.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.624+0000 7f3430ff9700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:53.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.624+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434045ad0 con 0x7f34340f6510 2023-11-15T05:07:53.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.625+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34280410a0 con 0x7f34340f6510 2023-11-15T05:07:53.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.625+0000 7f34337fe700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:53.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.625+0000 7f34337fe700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 0x7f341c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f34240096a0 tx=0x7f3424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:53.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.629+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f342801d7f0 con 0x7f34340f6510 2023-11-15T05:07:53.836 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.836+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34340fc050 con 0x7f34340f6510 2023-11-15T05:07:53.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.837+0000 7f3430ff9700 1 -- 172.21.15.154:0/883291767 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3428010230 con 0x7f34340f6510 2023-11-15T05:07:53.838 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:53.838 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:53.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 msgr2=0x7f341c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:53.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 0x7f341c032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f34240096a0 tx=0x7f3424006b40).stop 2023-11-15T05:07:53.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 msgr2=0x7f3434106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:53.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3428013510 tx=0x7f342800b040).stop 2023-11-15T05:07:53.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 shutdown_connections 2023-11-15T05:07:53.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:53.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 --2- 172.21.15.154:0/883291767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34340f6510 0x7f3434106730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:53.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 >> 172.21.15.154:0/883291767 conn(0x7f34340f1c70 msgr2=0x7f34340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:53.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 shutdown_connections 2023-11-15T05:07:53.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:53.838+0000 7f343a876700 1 -- 172.21.15.154:0/883291767 wait complete. 2023-11-15T05:07:53.842 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:53.837623+0000 mon.smithi086 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.154:0/883291767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:55.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:55.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:54.857602+0000 mgr.smithi086.hkfamo (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:56.384 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 -- 172.21.15.154:0/186177292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 msgr2=0x7f51700f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:56.384 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 --2- 172.21.15.154:0/186177292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51700f90f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5164009fb0 tx=0x7f5164009b30).stop 2023-11-15T05:07:56.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 -- 172.21.15.154:0/186177292 shutdown_connections 2023-11-15T05:07:56.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 --2- 172.21.15.154:0/186177292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51700f90f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:56.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 -- 172.21.15.154:0/186177292 >> 172.21.15.154:0/186177292 conn(0x7f51700f0b70 msgr2=0x7f51700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:56.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.382+0000 7f5176f18700 1 -- 172.21.15.154:0/186177292 shutdown_connections 2023-11-15T05:07:56.385 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.383+0000 7f5176f18700 1 -- 172.21.15.154:0/186177292 wait complete. 2023-11-15T05:07:56.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.383+0000 7f5176f18700 1 Processor -- start 2023-11-15T05:07:56.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.383+0000 7f5176f18700 1 -- start start 2023-11-15T05:07:56.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.383+0000 7f5176f18700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:56.386 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.383+0000 7f5176f18700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5170106bb0 con 0x7f5170060280 2023-11-15T05:07:56.388 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.384+0000 7f5174cb4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:56.388 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.384+0000 7f5174cb4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53858/0 (socket says 172.21.15.154:53858) 2023-11-15T05:07:56.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.384+0000 7f5174cb4700 1 -- 172.21.15.154:0/2519914984 learned_addr learned my addr 172.21.15.154:0/2519914984 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:56.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.384+0000 7f5174cb4700 1 -- 172.21.15.154:0/2519914984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5164005040 con 0x7f5170060280 2023-11-15T05:07:56.389 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.384+0000 7f5174cb4700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5164013510 tx=0x7f516400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:56.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5164009320 con 0x7f5170060280 2023-11-15T05:07:56.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5164013f00 con 0x7f5170060280 2023-11-15T05:07:56.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5170106db0 con 0x7f5170060280 2023-11-15T05:07:56.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5164014dc0 con 0x7f5170060280 2023-11-15T05:07:56.390 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5170107240 con 0x7f5170060280 2023-11-15T05:07:56.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5164014420 con 0x7f5170060280 2023-11-15T05:07:56.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.385+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5170045ad0 con 0x7f5170060280 2023-11-15T05:07:56.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.386+0000 7f516d7fa700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 0x7f5160032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:56.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.386+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51640410f0 con 0x7f5170060280 2023-11-15T05:07:56.391 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.386+0000 7f516ffff700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 0x7f5160032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:56.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.386+0000 7f516ffff700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 0x7f5160032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f515c0096a0 tx=0x7f515c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:56.392 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.390+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51640095f0 con 0x7f5170060280 2023-11-15T05:07:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.599+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51701080f0 con 0x7f5170060280 2023-11-15T05:07:56.601 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.600+0000 7f516d7fa700 1 -- 172.21.15.154:0/2519914984 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f516401cb10 con 0x7f5170060280 2023-11-15T05:07:56.602 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:56.602 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.601+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 msgr2=0x7f5160032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.601+0000 7f5176f18700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 0x7f5160032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f515c0096a0 tx=0x7f515c006b40).stop 2023-11-15T05:07:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.601+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 msgr2=0x7f51701066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:56.603 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.601+0000 7f5176f18700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5164013510 tx=0x7f516400b040).stop 2023-11-15T05:07:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 shutdown_connections 2023-11-15T05:07:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f516002ff60 0x7f5160032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 --2- 172.21.15.154:0/2519914984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5170060280 0x7f51701066c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 >> 172.21.15.154:0/2519914984 conn(0x7f51700f0b70 msgr2=0x7f51700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:56.604 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 shutdown_connections 2023-11-15T05:07:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:56.602+0000 7f5176f18700 1 -- 172.21.15.154:0/2519914984 wait complete. 2023-11-15T05:07:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:07:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:56.600517+0000 mon.smithi086 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.154:0/2519914984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:07:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:56.857922+0000 mgr.smithi086.hkfamo (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:58.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:07:58.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:07:59.192 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 -- 172.21.15.154:0/2151313736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 msgr2=0x7fec900f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 --2- 172.21.15.154:0/2151313736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec900f9190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fec80009fb0 tx=0x7fec80009b30).stop 2023-11-15T05:07:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 -- 172.21.15.154:0/2151313736 shutdown_connections 2023-11-15T05:07:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 --2- 172.21.15.154:0/2151313736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec900f9190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:59.193 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 -- 172.21.15.154:0/2151313736 >> 172.21.15.154:0/2151313736 conn(0x7fec900f0b70 msgr2=0x7fec900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 -- 172.21.15.154:0/2151313736 shutdown_connections 2023-11-15T05:07:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.190+0000 7fec97fee700 1 -- 172.21.15.154:0/2151313736 wait complete. 2023-11-15T05:07:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec97fee700 1 Processor -- start 2023-11-15T05:07:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec97fee700 1 -- start start 2023-11-15T05:07:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec97fee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:59.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec97fee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90106bd0 con 0x7fec900601e0 2023-11-15T05:07:59.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec95d8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:59.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec95d8a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53872/0 (socket says 172.21.15.154:53872) 2023-11-15T05:07:59.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.191+0000 7fec95d8a700 1 -- 172.21.15.154:0/2264319961 learned_addr learned my addr 172.21.15.154:0/2264319961 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:07:59.195 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec95d8a700 1 -- 172.21.15.154:0/2264319961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80005040 con 0x7fec900601e0 2023-11-15T05:07:59.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec95d8a700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec80004dd0 tx=0x7fec80009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:59.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80018070 con 0x7fec900601e0 2023-11-15T05:07:59.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90106dd0 con 0x7fec900601e0 2023-11-15T05:07:59.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec80013410 con 0x7fec900601e0 2023-11-15T05:07:59.196 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec8001c3c0 con 0x7fec900601e0 2023-11-15T05:07:59.197 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.192+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90107260 con 0x7fec900601e0 2023-11-15T05:07:59.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.193+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec8001d420 con 0x7fec900601e0 2023-11-15T05:07:59.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.193+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90045ad0 con 0x7fec900601e0 2023-11-15T05:07:59.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.193+0000 7fec867fc700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:07:59.198 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.193+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec800413d0 con 0x7fec900601e0 2023-11-15T05:07:59.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.193+0000 7fec95589700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:07:59.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.194+0000 7fec95589700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fec8c0096a0 tx=0x7fec8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:07:59.199 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.198+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec8001d820 con 0x7fec900601e0 2023-11-15T05:07:59.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:07:58.858244+0000 mgr.smithi086.hkfamo (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:07:59.403 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.402+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec901080b0 con 0x7fec900601e0 2023-11-15T05:07:59.403 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.403+0000 7fec867fc700 1 -- 172.21.15.154:0/2264319961 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec8001b370 con 0x7fec900601e0 2023-11-15T05:07:59.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:07:59.404 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:07:59.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 msgr2=0x7fec7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:59.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fec8c0096a0 tx=0x7fec8c006b40).stop 2023-11-15T05:07:59.405 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 msgr2=0x7fec901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:07:59.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec80004dd0 tx=0x7fec80009b30).stop 2023-11-15T05:07:59.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 shutdown_connections 2023-11-15T05:07:59.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:59.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 --2- 172.21.15.154:0/2264319961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec900601e0 0x7fec901066e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:07:59.406 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 >> 172.21.15.154:0/2264319961 conn(0x7fec900f0b70 msgr2=0x7fec900f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:07:59.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 shutdown_connections 2023-11-15T05:07:59.407 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:07:59.404+0000 7fec97fee700 1 -- 172.21.15.154:0/2264319961 wait complete. 2023-11-15T05:07:59.407 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:00.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:07:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:07:59.403885+0000 mon.smithi086 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.154:0/2264319961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:00.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:00.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:00.858591+0000 mgr.smithi086.hkfamo (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:01.949 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 -- 172.21.15.154:0/3726936380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 msgr2=0x7f236c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:01.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 --2- 172.21.15.154:0/3726936380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f2360009fb0 tx=0x7f2360009b30).stop 2023-11-15T05:08:01.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 -- 172.21.15.154:0/3726936380 shutdown_connections 2023-11-15T05:08:01.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 --2- 172.21.15.154:0/3726936380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:01.950 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 -- 172.21.15.154:0/3726936380 >> 172.21.15.154:0/3726936380 conn(0x7f236c0f0b70 msgr2=0x7f236c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 -- 172.21.15.154:0/3726936380 shutdown_connections 2023-11-15T05:08:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.946+0000 7f2371c12700 1 -- 172.21.15.154:0/3726936380 wait complete. 2023-11-15T05:08:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f2371c12700 1 Processor -- start 2023-11-15T05:08:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f2371c12700 1 -- start start 2023-11-15T05:08:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f2371c12700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:01.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f2371c12700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0f4ff0 con 0x7f236c060280 2023-11-15T05:08:01.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:01.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f236b7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53898/0 (socket says 172.21.15.154:53898) 2023-11-15T05:08:01.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.947+0000 7f236b7fe700 1 -- 172.21.15.154:0/4173580408 learned_addr learned my addr 172.21.15.154:0/4173580408 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:01.952 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f236b7fe700 1 -- 172.21.15.154:0/4173580408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360005040 con 0x7f236c060280 2023-11-15T05:08:01.953 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f236b7fe700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f2360009830 tx=0x7f236000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:01.953 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2360009320 con 0x7f236c060280 2023-11-15T05:08:01.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23600136c0 con 0x7f236c060280 2023-11-15T05:08:01.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f236001ee90 con 0x7f236c060280 2023-11-15T05:08:01.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c0f5190 con 0x7f236c060280 2023-11-15T05:08:01.954 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.948+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c0f5620 con 0x7f236c060280 2023-11-15T05:08:01.956 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.949+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f236001e420 con 0x7f236c060280 2023-11-15T05:08:01.958 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.949+0000 7f2353fff700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 0x7f23540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:01.958 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.949+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2360040ee0 con 0x7f236c060280 2023-11-15T05:08:01.958 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.949+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c059100 con 0x7f236c060280 2023-11-15T05:08:01.959 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.949+0000 7f236affd700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 0x7f23540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:01.959 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.950+0000 7f236affd700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 0x7f23540323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f235c0096a0 tx=0x7f235c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:01.959 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:01.954+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f236001c930 con 0x7f236c060280 2023-11-15T05:08:02.162 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.159+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f236c0f5c80 con 0x7f236c060280 2023-11-15T05:08:02.163 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.160+0000 7f2353fff700 1 -- 172.21.15.154:0/4173580408 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f236001cb40 con 0x7f236c060280 2023-11-15T05:08:02.163 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:02.163 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:02.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 msgr2=0x7f23540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:02.164 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 0x7f23540323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f235c0096a0 tx=0x7f235c006b40).stop 2023-11-15T05:08:02.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 msgr2=0x7f236c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:02.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f2360009830 tx=0x7f236000b040).stop 2023-11-15T05:08:02.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 shutdown_connections 2023-11-15T05:08:02.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f235402ff10 0x7f23540323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:02.165 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 --2- 172.21.15.154:0/4173580408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f236c060280 0x7f236c0f6b70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:02.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 >> 172.21.15.154:0/4173580408 conn(0x7f236c0f0b70 msgr2=0x7f236c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:02.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 shutdown_connections 2023-11-15T05:08:02.166 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:02.162+0000 7f2371c12700 1 -- 172.21.15.154:0/4173580408 wait complete. 2023-11-15T05:08:02.166 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:02.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:02.162231+0000 mon.smithi086 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.154:0/4173580408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:03.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:03.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:03.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:02.858903+0000 mgr.smithi086.hkfamo (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:04.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 -- 172.21.15.154:0/1094327916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 msgr2=0x7f12180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:04.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 --2- 172.21.15.154:0/1094327916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f12180f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f1214009fb0 tx=0x7f1214009b30).stop 2023-11-15T05:08:04.663 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 -- 172.21.15.154:0/1094327916 shutdown_connections 2023-11-15T05:08:04.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 --2- 172.21.15.154:0/1094327916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f12180f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:04.664 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 -- 172.21.15.154:0/1094327916 >> 172.21.15.154:0/1094327916 conn(0x7f12180f1c70 msgr2=0x7f12180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:04.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 -- 172.21.15.154:0/1094327916 shutdown_connections 2023-11-15T05:08:04.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.661+0000 7f121ff55700 1 -- 172.21.15.154:0/1094327916 wait complete. 2023-11-15T05:08:04.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121ff55700 1 Processor -- start 2023-11-15T05:08:04.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121ff55700 1 -- start start 2023-11-15T05:08:04.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121ff55700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:04.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121ff55700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1218106c20 con 0x7f12180f6510 2023-11-15T05:08:04.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121dcf1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:04.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121dcf1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59318/0 (socket says 172.21.15.154:59318) 2023-11-15T05:08:04.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121dcf1700 1 -- 172.21.15.154:0/2720780183 learned_addr learned my addr 172.21.15.154:0/2720780183 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:04.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121dcf1700 1 -- 172.21.15.154:0/2720780183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214005040 con 0x7f12180f6510 2023-11-15T05:08:04.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.662+0000 7f121dcf1700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f1214013510 tx=0x7f121400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:04.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1214009320 con 0x7f12180f6510 2023-11-15T05:08:04.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1214013f00 con 0x7f12180f6510 2023-11-15T05:08:04.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1218106e20 con 0x7f12180f6510 2023-11-15T05:08:04.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1214014d90 con 0x7f12180f6510 2023-11-15T05:08:04.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12181072b0 con 0x7f12180f6510 2023-11-15T05:08:04.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12140095f0 con 0x7f12180f6510 2023-11-15T05:08:04.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.663+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1218045ad0 con 0x7f12180f6510 2023-11-15T05:08:04.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.664+0000 7f120e7fc700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 0x7f1204032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:04.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.664+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1214041000 con 0x7f12180f6510 2023-11-15T05:08:04.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.664+0000 7f121d4f0700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 0x7f1204032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:04.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.664+0000 7f121d4f0700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 0x7f1204032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f12080096a0 tx=0x7f1208006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:04.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.669+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1214014420 con 0x7f12180f6510 2023-11-15T05:08:04.877 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.875+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12180fbf60 con 0x7f12180f6510 2023-11-15T05:08:04.877 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.876+0000 7f120e7fc700 1 -- 172.21.15.154:0/2720780183 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f121401d7f0 con 0x7f12180f6510 2023-11-15T05:08:04.878 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:04.878 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:04.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 msgr2=0x7f1204032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:04.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 0x7f1204032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f12080096a0 tx=0x7f1208006b40).stop 2023-11-15T05:08:04.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 msgr2=0x7f1218106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:04.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f1214013510 tx=0x7f121400b040).stop 2023-11-15T05:08:04.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 shutdown_connections 2023-11-15T05:08:04.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f120402ff60 0x7f1204032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:04.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 --2- 172.21.15.154:0/2720780183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12180f6510 0x7f1218106730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:04.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 >> 172.21.15.154:0/2720780183 conn(0x7f12180f1c70 msgr2=0x7f12180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:04.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 shutdown_connections 2023-11-15T05:08:04.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:04.877+0000 7f121ff55700 1 -- 172.21.15.154:0/2720780183 wait complete. 2023-11-15T05:08:04.882 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:04.859184+0000 mgr.smithi086.hkfamo (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:04.877924+0000 mon.smithi086 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.154:0/2720780183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:06.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:06.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:07.430 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 -- 172.21.15.154:0/70412644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 msgr2=0x7fd0380f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:07.430 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 --2- 172.21.15.154:0/70412644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0380f4b60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fd028009fb0 tx=0x7fd028009b30).stop 2023-11-15T05:08:07.430 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 -- 172.21.15.154:0/70412644 shutdown_connections 2023-11-15T05:08:07.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 --2- 172.21.15.154:0/70412644 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0380f4b60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:07.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 -- 172.21.15.154:0/70412644 >> 172.21.15.154:0/70412644 conn(0x7fd0380f0260 msgr2=0x7fd0380f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:07.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 -- 172.21.15.154:0/70412644 shutdown_connections 2023-11-15T05:08:07.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.427+0000 7fd03f36c700 1 -- 172.21.15.154:0/70412644 wait complete. 2023-11-15T05:08:07.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03f36c700 1 Processor -- start 2023-11-15T05:08:07.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03f36c700 1 -- start start 2023-11-15T05:08:07.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03f36c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:07.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03f36c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038106bc0 con 0x7fd0380f4740 2023-11-15T05:08:07.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03d108700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03d108700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59334/0 (socket says 172.21.15.154:59334) 2023-11-15T05:08:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03d108700 1 -- 172.21.15.154:0/1623017195 learned_addr learned my addr 172.21.15.154:0/1623017195 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.428+0000 7fd03d108700 1 -- 172.21.15.154:0/1623017195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd028005040 con 0x7fd0380f4740 2023-11-15T05:08:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd03d108700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd028013510 tx=0x7fd02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd028009320 con 0x7fd0380f4740 2023-11-15T05:08:07.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd038106dc0 con 0x7fd0380f4740 2023-11-15T05:08:07.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd028013f00 con 0x7fd0380f4740 2023-11-15T05:08:07.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd028014d90 con 0x7fd0380f4740 2023-11-15T05:08:07.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.429+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd038107250 con 0x7fd0380f4740 2023-11-15T05:08:07.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.430+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd038045ad0 con 0x7fd0380f4740 2023-11-15T05:08:07.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.430+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd028014420 con 0x7fd0380f4740 2023-11-15T05:08:07.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.430+0000 7fd02dffb700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:07.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.430+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd028041080 con 0x7fd0380f4740 2023-11-15T05:08:07.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.430+0000 7fd03c907700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:07.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.431+0000 7fd03c907700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 0x7fd024032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd0340096a0 tx=0x7fd034006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:07.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.435+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0280095f0 con 0x7fd0380f4740 2023-11-15T05:08:07.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.640+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd038108100 con 0x7fd0380f4740 2023-11-15T05:08:07.641 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.641+0000 7fd02dffb700 1 -- 172.21.15.154:0/1623017195 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd02801db70 con 0x7fd0380f4740 2023-11-15T05:08:07.642 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:07.642 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:07.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 msgr2=0x7fd024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:07.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 0x7fd024032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fd0340096a0 tx=0x7fd034006b40).stop 2023-11-15T05:08:07.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 msgr2=0x7fd0381066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:07.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fd028013510 tx=0x7fd02800b040).stop 2023-11-15T05:08:07.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 shutdown_connections 2023-11-15T05:08:07.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd02402ff60 0x7fd024032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:07.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 --2- 172.21.15.154:0/1623017195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0380f4740 0x7fd0381066d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:07.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 >> 172.21.15.154:0/1623017195 conn(0x7fd0380f0260 msgr2=0x7fd0380f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:07.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 shutdown_connections 2023-11-15T05:08:07.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:07.643+0000 7fd03f36c700 1 -- 172.21.15.154:0/1623017195 wait complete. 2023-11-15T05:08:07.646 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:06.859580+0000 mgr.smithi086.hkfamo (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:08.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:07.641904+0000 mon.smithi086 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.154:0/1623017195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:09.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:09.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:09.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:08.859969+0000 mgr.smithi086.hkfamo (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:10.272 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- 172.21.15.154:0/3540237027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 msgr2=0x7f54240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:10.273 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 --2- 172.21.15.154:0/3540237027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54240f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5414009fb0 tx=0x7f5414009b30).stop 2023-11-15T05:08:10.273 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- 172.21.15.154:0/3540237027 shutdown_connections 2023-11-15T05:08:10.273 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 --2- 172.21.15.154:0/3540237027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54240f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:10.273 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- 172.21.15.154:0/3540237027 >> 172.21.15.154:0/3540237027 conn(0x7f54240f09a0 msgr2=0x7f54240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:10.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- 172.21.15.154:0/3540237027 shutdown_connections 2023-11-15T05:08:10.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- 172.21.15.154:0/3540237027 wait complete. 2023-11-15T05:08:10.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 Processor -- start 2023-11-15T05:08:10.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.271+0000 7f542359e700 1 -- start start 2023-11-15T05:08:10.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542359e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:10.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542359e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424106990 con 0x7f54240f51a0 2023-11-15T05:08:10.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:10.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59364/0 (socket says 172.21.15.154:59364) 2023-11-15T05:08:10.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542259c700 1 -- 172.21.15.154:0/2981306385 learned_addr learned my addr 172.21.15.154:0/2981306385 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:10.275 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542259c700 1 -- 172.21.15.154:0/2981306385 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414005040 con 0x7f54240f51a0 2023-11-15T05:08:10.276 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542259c700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5414013510 tx=0x7f541400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:10.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5414009320 con 0x7f54240f51a0 2023-11-15T05:08:10.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424106b90 con 0x7f54240f51a0 2023-11-15T05:08:10.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5414013f00 con 0x7f54240f51a0 2023-11-15T05:08:10.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5414014dc0 con 0x7f54240f51a0 2023-11-15T05:08:10.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.272+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5424107020 con 0x7f54240f51a0 2023-11-15T05:08:10.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.273+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54140095f0 con 0x7f54240f51a0 2023-11-15T05:08:10.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.273+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424045ad0 con 0x7f54240f51a0 2023-11-15T05:08:10.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.273+0000 7f5412ffd700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 0x7f540c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:10.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.273+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5414041080 con 0x7f54240f51a0 2023-11-15T05:08:10.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.274+0000 7f5421d9b700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 0x7f540c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:10.279 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.274+0000 7f5421d9b700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 0x7f540c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f54180096a0 tx=0x7f5418006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:10.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.278+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f541401c790 con 0x7f54240f51a0 2023-11-15T05:08:10.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.479+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5424107ed0 con 0x7f54240f51a0 2023-11-15T05:08:10.481 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.480+0000 7f5412ffd700 1 -- 172.21.15.154:0/2981306385 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5414010230 con 0x7f54240f51a0 2023-11-15T05:08:10.482 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:10.482 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:10.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 msgr2=0x7f540c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:10.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 0x7f540c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f54180096a0 tx=0x7f5418006b40).stop 2023-11-15T05:08:10.483 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 msgr2=0x7f54241064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:10.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5414013510 tx=0x7f541400b040).stop 2023-11-15T05:08:10.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 shutdown_connections 2023-11-15T05:08:10.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f540c02ff60 0x7f540c032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:10.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 --2- 172.21.15.154:0/2981306385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54240f51a0 0x7f54241064a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:10.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 >> 172.21.15.154:0/2981306385 conn(0x7f54240f09a0 msgr2=0x7f542405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:10.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 shutdown_connections 2023-11-15T05:08:10.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:10.482+0000 7f542359e700 1 -- 172.21.15.154:0/2981306385 wait complete. 2023-11-15T05:08:10.486 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:10.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:10.481135+0000 mon.smithi086 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.154:0/2981306385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:11.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:11.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:11.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:11 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:10.860334+0000 mgr.smithi086.hkfamo (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:13.013 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 -- 172.21.15.154:0/1227801119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 msgr2=0x7f2bc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:13.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/1227801119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc80f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f2bb8009fb0 tx=0x7f2bb8009b30).stop 2023-11-15T05:08:13.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 -- 172.21.15.154:0/1227801119 shutdown_connections 2023-11-15T05:08:13.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/1227801119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc80f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:13.014 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 -- 172.21.15.154:0/1227801119 >> 172.21.15.154:0/1227801119 conn(0x7f2bc80f0b70 msgr2=0x7f2bc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 -- 172.21.15.154:0/1227801119 shutdown_connections 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 -- 172.21.15.154:0/1227801119 wait complete. 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.011+0000 7f2bccaa2700 1 Processor -- start 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bccaa2700 1 -- start start 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bccaa2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:13.015 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bccaa2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc8060220 con 0x7f2bc80f5410 2023-11-15T05:08:13.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bc659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:13.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bc659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:59376/0 (socket says 172.21.15.154:59376) 2023-11-15T05:08:13.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bc659c700 1 -- 172.21.15.154:0/3965546246 learned_addr learned my addr 172.21.15.154:0/3965546246 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:13.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bc659c700 1 -- 172.21.15.154:0/3965546246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb8005040 con 0x7f2bc80f5410 2023-11-15T05:08:13.016 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bc659c700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f2bb8013510 tx=0x7f2bb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb8009320 con 0x7f2bc80f5410 2023-11-15T05:08:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc8060420 con 0x7f2bc80f5410 2023-11-15T05:08:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.012+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb8013f00 con 0x7f2bc80f5410 2023-11-15T05:08:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.013+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb8014d90 con 0x7f2bc80f5410 2023-11-15T05:08:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.013+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc8109b00 con 0x7f2bc80f5410 2023-11-15T05:08:13.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.013+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bb80095f0 con 0x7f2bc80f5410 2023-11-15T05:08:13.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.013+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc8059100 con 0x7f2bc80f5410 2023-11-15T05:08:13.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.014+0000 7f2bb6ffd700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 0x7f2bb0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:13.019 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.014+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bb8040fe0 con 0x7f2bc80f5410 2023-11-15T05:08:13.020 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.014+0000 7f2bc5d9b700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 0x7f2bb0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:13.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.014+0000 7f2bc5d9b700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 0x7f2bb0032470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2bbc0096a0 tx=0x7f2bbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:13.021 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.018+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bb806d2b0 con 0x7f2bc80f5410 2023-11-15T05:08:13.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.223+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bc8045ad0 con 0x7f2bc80f5410 2023-11-15T05:08:13.226 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.224+0000 7f2bb6ffd700 1 -- 172.21.15.154:0/3965546246 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bb8014420 con 0x7f2bc80f5410 2023-11-15T05:08:13.227 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:13.227 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:13.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 msgr2=0x7f2bb0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:13.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 0x7f2bb0032470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2bbc0096a0 tx=0x7f2bbc006b40).stop 2023-11-15T05:08:13.228 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 msgr2=0x7f2bc8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:13.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f2bb8013510 tx=0x7f2bb800b040).stop 2023-11-15T05:08:13.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 shutdown_connections 2023-11-15T05:08:13.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f2bb002ffb0 0x7f2bb0032470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:13.229 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 --2- 172.21.15.154:0/3965546246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc80f5410 0x7f2bc8109450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:13.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 >> 172.21.15.154:0/3965546246 conn(0x7f2bc80f0b70 msgr2=0x7f2bc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:13.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 shutdown_connections 2023-11-15T05:08:13.230 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:13.226+0000 7f2bccaa2700 1 -- 172.21.15.154:0/3965546246 wait complete. 2023-11-15T05:08:13.230 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:13.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:12.860730+0000 mgr.smithi086.hkfamo (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:14.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:13.225276+0000 mon.smithi086 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.154:0/3965546246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:14.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:14.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:15.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:14.861056+0000 mgr.smithi086.hkfamo (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:15.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.838+0000 7f04ff782700 1 -- 172.21.15.154:0/563999844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 msgr2=0x7f04f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:15.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 --2- 172.21.15.154:0/563999844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f80f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f04e8009fb0 tx=0x7f04e8009b30).stop 2023-11-15T05:08:15.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- 172.21.15.154:0/563999844 shutdown_connections 2023-11-15T05:08:15.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 --2- 172.21.15.154:0/563999844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f80f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:15.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- 172.21.15.154:0/563999844 >> 172.21.15.154:0/563999844 conn(0x7f04f80f1c70 msgr2=0x7f04f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:15.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- 172.21.15.154:0/563999844 shutdown_connections 2023-11-15T05:08:15.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- 172.21.15.154:0/563999844 wait complete. 2023-11-15T05:08:15.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 Processor -- start 2023-11-15T05:08:15.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- start start 2023-11-15T05:08:15.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:15.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.839+0000 7f04ff782700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f8106c20 con 0x7f04f80f6510 2023-11-15T05:08:15.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04fd51e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:15.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04fd51e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37010/0 (socket says 172.21.15.154:37010) 2023-11-15T05:08:15.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04fd51e700 1 -- 172.21.15.154:0/2336383099 learned_addr learned my addr 172.21.15.154:0/2336383099 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:15.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04fd51e700 1 -- 172.21.15.154:0/2336383099 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e8005040 con 0x7f04f80f6510 2023-11-15T05:08:15.844 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04fd51e700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f04e8013510 tx=0x7f04e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e8009320 con 0x7f04f80f6510 2023-11-15T05:08:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f8106e20 con 0x7f04f80f6510 2023-11-15T05:08:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04e8013f00 con 0x7f04f80f6510 2023-11-15T05:08:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.840+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e8014d90 con 0x7f04f80f6510 2023-11-15T05:08:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.841+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f81072b0 con 0x7f04f80f6510 2023-11-15T05:08:15.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.841+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04e80263a0 con 0x7f04f80f6510 2023-11-15T05:08:15.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.842+0000 7f04edffb700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 0x7f04e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:15.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.842+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04e8041070 con 0x7f04f80f6510 2023-11-15T05:08:15.846 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.842+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f8045ad0 con 0x7f04f80f6510 2023-11-15T05:08:15.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.842+0000 7f04fcd1d700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 0x7f04e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:15.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.842+0000 7f04fcd1d700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 0x7f04e4032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f04f40096a0 tx=0x7f04f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:15.847 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:15.846+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04e8012920 con 0x7f04f80f6510 2023-11-15T05:08:16.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.054+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04f80fc050 con 0x7f04f80f6510 2023-11-15T05:08:16.055 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.055+0000 7f04edffb700 1 -- 172.21.15.154:0/2336383099 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04e800f340 con 0x7f04f80f6510 2023-11-15T05:08:16.056 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:16.056 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:16.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 msgr2=0x7f04e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:16.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 0x7f04e4032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f04f40096a0 tx=0x7f04f4006b40).stop 2023-11-15T05:08:16.057 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 msgr2=0x7f04f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:16.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f04e8013510 tx=0x7f04e800b040).stop 2023-11-15T05:08:16.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 shutdown_connections 2023-11-15T05:08:16.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f04e402ffb0 0x7f04e4032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:16.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 --2- 172.21.15.154:0/2336383099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04f80f6510 0x7f04f8106730 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:16.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 >> 172.21.15.154:0/2336383099 conn(0x7f04f80f1c70 msgr2=0x7f04f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:16.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 shutdown_connections 2023-11-15T05:08:16.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:16.056+0000 7f04ff782700 1 -- 172.21.15.154:0/2336383099 wait complete. 2023-11-15T05:08:16.059 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:16.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:16 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:16.055792+0000 mon.smithi086 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.154:0/2336383099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:17.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:17.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:16.861429+0000 mgr.smithi086.hkfamo (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:18.719 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3494377984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 msgr2=0x7fbcd40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:18.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3494377984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd40f9150 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fbcbc009fb0 tx=0x7fbcbc009b30).stop 2023-11-15T05:08:18.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3494377984 shutdown_connections 2023-11-15T05:08:18.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3494377984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd40f9150 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:18.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3494377984 >> 172.21.15.154:0/3494377984 conn(0x7fbcd40f0b70 msgr2=0x7fbcd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:18.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3494377984 shutdown_connections 2023-11-15T05:08:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.717+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3494377984 wait complete. 2023-11-15T05:08:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd8ba9700 1 Processor -- start 2023-11-15T05:08:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd8ba9700 1 -- start start 2023-11-15T05:08:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd8ba9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd8ba9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4106ba0 con 0x7fbcd4060220 2023-11-15T05:08:18.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:18.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37030/0 (socket says 172.21.15.154:37030) 2023-11-15T05:08:18.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.718+0000 7fbcd259c700 1 -- 172.21.15.154:0/3030708351 learned_addr learned my addr 172.21.15.154:0/3030708351 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:18.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbcd259c700 1 -- 172.21.15.154:0/3030708351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc005040 con 0x7fbcd4060220 2023-11-15T05:08:18.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbcd259c700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fbcbc013510 tx=0x7fbcbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:18.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcbc009320 con 0x7fbcd4060220 2023-11-15T05:08:18.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd4106da0 con 0x7fbcd4060220 2023-11-15T05:08:18.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcbc013f00 con 0x7fbcd4060220 2023-11-15T05:08:18.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcbc014dc0 con 0x7fbcd4060220 2023-11-15T05:08:18.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.719+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd4107230 con 0x7fbcd4060220 2023-11-15T05:08:18.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.720+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcbc0095f0 con 0x7fbcd4060220 2023-11-15T05:08:18.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.720+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd4045ad0 con 0x7fbcd4060220 2023-11-15T05:08:18.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.720+0000 7fbccaffd700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 0x7fbcc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:18.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.720+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcbc041030 con 0x7fbcd4060220 2023-11-15T05:08:18.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.720+0000 7fbcd1d9b700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 0x7fbcc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:18.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.721+0000 7fbcd1d9b700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 0x7fbcc0032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:18.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.725+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcbc014420 con 0x7fbcd4060220 2023-11-15T05:08:18.930 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.930+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcd41080e0 con 0x7fbcd4060220 2023-11-15T05:08:18.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.930+0000 7fbccaffd700 1 -- 172.21.15.154:0/3030708351 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcbc010230 con 0x7fbcd4060220 2023-11-15T05:08:18.931 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:18.931 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:18.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 msgr2=0x7fbcc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:18.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 0x7fbcc0032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).stop 2023-11-15T05:08:18.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 msgr2=0x7fbcd41066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:18.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fbcbc013510 tx=0x7fbcbc00b040).stop 2023-11-15T05:08:18.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 shutdown_connections 2023-11-15T05:08:18.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbcc002ff60 0x7fbcc0032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:18.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 --2- 172.21.15.154:0/3030708351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd4060220 0x7fbcd41066b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:18.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 >> 172.21.15.154:0/3030708351 conn(0x7fbcd40f0b70 msgr2=0x7fbcd40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:18.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 shutdown_connections 2023-11-15T05:08:18.935 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:18.932+0000 7fbcd8ba9700 1 -- 172.21.15.154:0/3030708351 wait complete. 2023-11-15T05:08:18.936 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:18.861787+0000 mgr.smithi086.hkfamo (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:20.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:18.931339+0000 mon.smithi086 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.154:0/3030708351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:20.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:20.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:20.862093+0000 mgr.smithi086.hkfamo (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:21.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.441+0000 7fa4fc61c700 1 -- 172.21.15.154:0/3288003874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 msgr2=0x7fa4f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:21.444 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.441+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/3288003874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f40f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fa4e4009fb0 tx=0x7fa4e4009b30).stop 2023-11-15T05:08:21.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.441+0000 7fa4fc61c700 1 -- 172.21.15.154:0/3288003874 shutdown_connections 2023-11-15T05:08:21.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.441+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/3288003874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f40f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:21.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.441+0000 7fa4fc61c700 1 -- 172.21.15.154:0/3288003874 >> 172.21.15.154:0/3288003874 conn(0x7fa4f40f0b70 msgr2=0x7fa4f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:21.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 -- 172.21.15.154:0/3288003874 shutdown_connections 2023-11-15T05:08:21.445 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 -- 172.21.15.154:0/3288003874 wait complete. 2023-11-15T05:08:21.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 Processor -- start 2023-11-15T05:08:21.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 -- start start 2023-11-15T05:08:21.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:21.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.442+0000 7fa4fc61c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4060240 con 0x7fa4f40f5410 2023-11-15T05:08:21.446 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fa3b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:21.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fa3b8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37044/0 (socket says 172.21.15.154:37044) 2023-11-15T05:08:21.447 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fa3b8700 1 -- 172.21.15.154:0/698944665 learned_addr learned my addr 172.21.15.154:0/698944665 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:21.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fa3b8700 1 -- 172.21.15.154:0/698944665 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4005040 con 0x7fa4f40f5410 2023-11-15T05:08:21.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fa3b8700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fa4e4014000 tx=0x7fa4e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:21.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e40095f0 con 0x7fa4f40f5410 2023-11-15T05:08:21.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f4060440 con 0x7fa4f40f5410 2023-11-15T05:08:21.448 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4e4007e70 con 0x7fa4f40f5410 2023-11-15T05:08:21.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e4014ea0 con 0x7fa4f40f5410 2023-11-15T05:08:21.449 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.443+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f4109ae0 con 0x7fa4f40f5410 2023-11-15T05:08:21.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.444+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4e4014690 con 0x7fa4f40f5410 2023-11-15T05:08:21.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.444+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f4045ad0 con 0x7fa4f40f5410 2023-11-15T05:08:21.450 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.444+0000 7fa4eaffd700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:21.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.444+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4e4040930 con 0x7fa4f40f5410 2023-11-15T05:08:21.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.445+0000 7fa4f9bb7700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:21.451 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.445+0000 7fa4f9bb7700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 0x7fa4e0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa4f00096a0 tx=0x7fa4f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:21.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.449+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4e400f300 con 0x7fa4f40f5410 2023-11-15T05:08:21.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.657+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4f40fae60 con 0x7fa4f40f5410 2023-11-15T05:08:21.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.657+0000 7fa4eaffd700 1 -- 172.21.15.154:0/698944665 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4e400f300 con 0x7fa4f40f5410 2023-11-15T05:08:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:21.659 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:21.659 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 msgr2=0x7fa4e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 0x7fa4e0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa4f00096a0 tx=0x7fa4f0006b40).stop 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 msgr2=0x7fa4f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fa4e4014000 tx=0x7fa4e400b040).stop 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 shutdown_connections 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 --2- 172.21.15.154:0/698944665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:21.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 >> 172.21.15.154:0/698944665 conn(0x7fa4f40f0b70 msgr2=0x7fa4f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:21.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 shutdown_connections 2023-11-15T05:08:21.661 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:21.659+0000 7fa4fc61c700 1 -- 172.21.15.154:0/698944665 wait complete. 2023-11-15T05:08:21.662 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:22.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:21.658417+0000 mon.smithi086 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.154:0/698944665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:23.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:23.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:23.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:22.862476+0000 mgr.smithi086.hkfamo (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:24.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 -- 172.21.15.154:0/3084609606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 msgr2=0x7f0c100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:24.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/3084609606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c100f90f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f0c00009fb0 tx=0x7f0c00009b30).stop 2023-11-15T05:08:24.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 -- 172.21.15.154:0/3084609606 shutdown_connections 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/3084609606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c100f90f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 -- 172.21.15.154:0/3084609606 >> 172.21.15.154:0/3084609606 conn(0x7f0c100f0b70 msgr2=0x7f0c100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 -- 172.21.15.154:0/3084609606 shutdown_connections 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.203+0000 7f0c15f5f700 1 -- 172.21.15.154:0/3084609606 wait complete. 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c15f5f700 1 Processor -- start 2023-11-15T05:08:24.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c15f5f700 1 -- start start 2023-11-15T05:08:24.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c15f5f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:24.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c15f5f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c10106bd0 con 0x7f0c10060280 2023-11-15T05:08:24.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c0f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:24.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c0f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55326/0 (socket says 172.21.15.154:55326) 2023-11-15T05:08:24.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c0f7fe700 1 -- 172.21.15.154:0/1471795201 learned_addr learned my addr 172.21.15.154:0/1471795201 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:24.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c0f7fe700 1 -- 172.21.15.154:0/1471795201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00005040 con 0x7f0c10060280 2023-11-15T05:08:24.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.204+0000 7f0c0f7fe700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f0c00013510 tx=0x7f0c0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:24.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00009320 con 0x7f0c10060280 2023-11-15T05:08:24.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c10106dd0 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c00013f00 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00014dc0 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c10107260 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.205+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c0001d3d0 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.206+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c10045ad0 con 0x7f0c10060280 2023-11-15T05:08:24.211 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.206+0000 7f0c07fff700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:24.212 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.206+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c00041350 con 0x7f0c10060280 2023-11-15T05:08:24.212 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.206+0000 7f0c0effd700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:24.212 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.206+0000 7f0c0effd700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 0x7f0bfc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0bf80096a0 tx=0x7f0bf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:24.213 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.210+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c00012510 con 0x7f0c10060280 2023-11-15T05:08:24.420 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.419+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c10108110 con 0x7f0c10060280 2023-11-15T05:08:24.420 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.420+0000 7f0c07fff700 1 -- 172.21.15.154:0/1471795201 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c00012330 con 0x7f0c10060280 2023-11-15T05:08:24.421 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:24.421 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:24.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.421+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 msgr2=0x7f0bfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:24.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.421+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 0x7f0bfc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0bf80096a0 tx=0x7f0bf8006b40).stop 2023-11-15T05:08:24.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.421+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 msgr2=0x7f0c101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:24.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.421+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f0c00013510 tx=0x7f0c0000b040).stop 2023-11-15T05:08:24.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 shutdown_connections 2023-11-15T05:08:24.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0bfc02ff60 0x7f0bfc032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:24.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 --2- 172.21.15.154:0/1471795201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c10060280 0x7f0c101066e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:24.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 >> 172.21.15.154:0/1471795201 conn(0x7f0c100f0b70 msgr2=0x7f0c100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:24.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 shutdown_connections 2023-11-15T05:08:24.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:24.422+0000 7f0c15f5f700 1 -- 172.21.15.154:0/1471795201 wait complete. 2023-11-15T05:08:24.425 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:24.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:24.420962+0000 mon.smithi086 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.154:0/1471795201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:25.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:24.862725+0000 mgr.smithi086.hkfamo (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:25.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:24.932481+0000 mon.smithi086 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:08:25.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:24.938890+0000 mon.smithi086 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:08:25.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:25.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:26.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 -- 172.21.15.154:0/2905944106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 msgr2=0x7f39900f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:26.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 --2- 172.21.15.154:0/2905944106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39900f4c00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f3984009fb0 tx=0x7f3984009b30).stop 2023-11-15T05:08:26.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 -- 172.21.15.154:0/2905944106 shutdown_connections 2023-11-15T05:08:26.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 --2- 172.21.15.154:0/2905944106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39900f4c00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:26.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 -- 172.21.15.154:0/2905944106 >> 172.21.15.154:0/2905944106 conn(0x7f39900f0260 msgr2=0x7f39900f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:26.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 -- 172.21.15.154:0/2905944106 shutdown_connections 2023-11-15T05:08:26.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.927+0000 7f3995355700 1 -- 172.21.15.154:0/2905944106 wait complete. 2023-11-15T05:08:26.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.928+0000 7f3995355700 1 Processor -- start 2023-11-15T05:08:26.931 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.928+0000 7f3995355700 1 -- start start 2023-11-15T05:08:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f3995355700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f3995355700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3990106dc0 con 0x7f39900f47e0 2023-11-15T05:08:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f398effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f398effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55338/0 (socket says 172.21.15.154:55338) 2023-11-15T05:08:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f398effd700 1 -- 172.21.15.154:0/4115466183 learned_addr learned my addr 172.21.15.154:0/4115466183 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:26.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f398effd700 1 -- 172.21.15.154:0/4115466183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3984005040 con 0x7f39900f47e0 2023-11-15T05:08:26.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f398effd700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3984013510 tx=0x7f398400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:26.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3984009320 con 0x7f39900f47e0 2023-11-15T05:08:26.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3990106fc0 con 0x7f39900f47e0 2023-11-15T05:08:26.933 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3984013f00 con 0x7f39900f47e0 2023-11-15T05:08:26.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.929+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3984014dc0 con 0x7f39900f47e0 2023-11-15T05:08:26.934 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.930+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3990107450 con 0x7f39900f47e0 2023-11-15T05:08:26.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.930+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3984014420 con 0x7f39900f47e0 2023-11-15T05:08:26.937 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.930+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990045ad0 con 0x7f39900f47e0 2023-11-15T05:08:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.930+0000 7f39777fe700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 0x7f3978032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.930+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3984041110 con 0x7f39900f47e0 2023-11-15T05:08:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.931+0000 7f398e7fc700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 0x7f3978032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.931+0000 7f398e7fc700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 0x7f3978032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f39800096a0 tx=0x7f3980006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:26.939 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:26.935+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39840095f0 con 0x7f39900f47e0 2023-11-15T05:08:27.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.141+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f399005eb50 con 0x7f39900f47e0 2023-11-15T05:08:27.142 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.141+0000 7f39777fe700 1 -- 172.21.15.154:0/4115466183 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f398401cb10 con 0x7f39900f47e0 2023-11-15T05:08:27.143 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:27.143 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:27.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 msgr2=0x7f3978032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:27.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 0x7f3978032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f39800096a0 tx=0x7f3980006b40).stop 2023-11-15T05:08:27.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 msgr2=0x7f39901068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:27.144 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3984013510 tx=0x7f398400b040).stop 2023-11-15T05:08:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 shutdown_connections 2023-11-15T05:08:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f397802ff60 0x7f3978032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 --2- 172.21.15.154:0/4115466183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39900f47e0 0x7f39901068d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 >> 172.21.15.154:0/4115466183 conn(0x7f39900f0260 msgr2=0x7f39900f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 shutdown_connections 2023-11-15T05:08:27.146 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:27.143+0000 7f3995355700 1 -- 172.21.15.154:0/4115466183 wait complete. 2023-11-15T05:08:27.146 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:26.862973+0000 mgr.smithi086.hkfamo (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:27.142438+0000 mon.smithi086 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.154:0/4115466183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:28.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:28.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:29.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:28.863377+0000 mgr.smithi086.hkfamo (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:29.753 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.751+0000 7f53c267d700 1 -- 172.21.15.154:0/4124235774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 msgr2=0x7f53bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.751+0000 7f53c267d700 1 --2- 172.21.15.154:0/4124235774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc0f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f53ac009fb0 tx=0x7f53ac009b30).stop 2023-11-15T05:08:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- 172.21.15.154:0/4124235774 shutdown_connections 2023-11-15T05:08:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 --2- 172.21.15.154:0/4124235774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc0f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:29.754 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- 172.21.15.154:0/4124235774 >> 172.21.15.154:0/4124235774 conn(0x7f53bc0f09a0 msgr2=0x7f53bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- 172.21.15.154:0/4124235774 shutdown_connections 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- 172.21.15.154:0/4124235774 wait complete. 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 Processor -- start 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- start start 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.752+0000 7f53c267d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc060240 con 0x7f53bc0f51a0 2023-11-15T05:08:29.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c167b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:29.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c167b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55356/0 (socket says 172.21.15.154:55356) 2023-11-15T05:08:29.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c167b700 1 -- 172.21.15.154:0/3719156248 learned_addr learned my addr 172.21.15.154:0/3719156248 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:29.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c167b700 1 -- 172.21.15.154:0/3719156248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac005040 con 0x7f53bc0f51a0 2023-11-15T05:08:29.756 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c167b700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f53ac013510 tx=0x7f53ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:29.758 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ac009320 con 0x7f53bc0f51a0 2023-11-15T05:08:29.758 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc060440 con 0x7f53bc0f51a0 2023-11-15T05:08:29.758 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53ac013f00 con 0x7f53bc0f51a0 2023-11-15T05:08:29.758 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.753+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53ac014dc0 con 0x7f53bc0f51a0 2023-11-15T05:08:29.759 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.754+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc109860 con 0x7f53bc0f51a0 2023-11-15T05:08:29.759 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.754+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53ac0095f0 con 0x7f53bc0f51a0 2023-11-15T05:08:29.759 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.754+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc045ad0 con 0x7f53bc0f51a0 2023-11-15T05:08:29.759 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.755+0000 7f53b1ffb700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 0x7f53a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:29.759 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.755+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53ac041040 con 0x7f53bc0f51a0 2023-11-15T05:08:29.760 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.755+0000 7f53c0e7a700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 0x7f53a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:29.760 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.755+0000 7f53c0e7a700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 0x7f53a8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f53b80096a0 tx=0x7f53b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:29.760 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.760+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53ac014420 con 0x7f53bc0f51a0 2023-11-15T05:08:29.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.963+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53bc10a080 con 0x7f53bc0f51a0 2023-11-15T05:08:29.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.964+0000 7f53b1ffb700 1 -- 172.21.15.154:0/3719156248 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f53ac010230 con 0x7f53bc0f51a0 2023-11-15T05:08:29.965 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:29.966 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:29.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 msgr2=0x7f53a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:29.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 0x7f53a8032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f53b80096a0 tx=0x7f53b8006b40).stop 2023-11-15T05:08:29.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 msgr2=0x7f53bc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:29.967 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f53ac013510 tx=0x7f53ac00b040).stop 2023-11-15T05:08:29.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 shutdown_connections 2023-11-15T05:08:29.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f53a802ff60 0x7f53a8032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:29.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 --2- 172.21.15.154:0/3719156248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53bc0f51a0 0x7f53bc1091b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:29.968 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 >> 172.21.15.154:0/3719156248 conn(0x7f53bc0f09a0 msgr2=0x7f53bc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:29.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 shutdown_connections 2023-11-15T05:08:29.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:29.966+0000 7f53c267d700 1 -- 172.21.15.154:0/3719156248 wait complete. 2023-11-15T05:08:29.969 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:30.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:29.965034+0000 mon.smithi086 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.154:0/3719156248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:31.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:31.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:31.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:30.863785+0000 mgr.smithi086.hkfamo (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:32.510 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 -- 172.21.15.154:0/1167715810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 msgr2=0x7f86400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:32.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 --2- 172.21.15.154:0/1167715810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86400f90f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f8634009fb0 tx=0x7f8634009b30).stop 2023-11-15T05:08:32.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 -- 172.21.15.154:0/1167715810 shutdown_connections 2023-11-15T05:08:32.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 --2- 172.21.15.154:0/1167715810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86400f90f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:32.511 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 -- 172.21.15.154:0/1167715810 >> 172.21.15.154:0/1167715810 conn(0x7f86400f0b70 msgr2=0x7f86400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:32.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 -- 172.21.15.154:0/1167715810 shutdown_connections 2023-11-15T05:08:32.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.509+0000 7f8645abf700 1 -- 172.21.15.154:0/1167715810 wait complete. 2023-11-15T05:08:32.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f8645abf700 1 Processor -- start 2023-11-15T05:08:32.512 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f8645abf700 1 -- start start 2023-11-15T05:08:32.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f8645abf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:32.513 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f8645abf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8640106bd0 con 0x7f8640060280 2023-11-15T05:08:32.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f863f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f863f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55366/0 (socket says 172.21.15.154:55366) 2023-11-15T05:08:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f863f7fe700 1 -- 172.21.15.154:0/1102098002 learned_addr learned my addr 172.21.15.154:0/1102098002 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.510+0000 7f863f7fe700 1 -- 172.21.15.154:0/1102098002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634005040 con 0x7f8640060280 2023-11-15T05:08:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f863f7fe700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8634013510 tx=0x7f863400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8634009320 con 0x7f8640060280 2023-11-15T05:08:32.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8640106dd0 con 0x7f8640060280 2023-11-15T05:08:32.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8634013f00 con 0x7f8640060280 2023-11-15T05:08:32.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8634014dc0 con 0x7f8640060280 2023-11-15T05:08:32.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.511+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8640107260 con 0x7f8640060280 2023-11-15T05:08:32.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.512+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86340095f0 con 0x7f8640060280 2023-11-15T05:08:32.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.512+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8640045ad0 con 0x7f8640060280 2023-11-15T05:08:32.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.512+0000 7f8627fff700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 0x7f8628032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:32.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.512+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8634041000 con 0x7f8640060280 2023-11-15T05:08:32.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.512+0000 7f863effd700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 0x7f8628032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:32.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.513+0000 7f863effd700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 0x7f8628032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86300096a0 tx=0x7f8630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:32.518 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.517+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8634014420 con 0x7f8640060280 2023-11-15T05:08:32.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.722+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8640108110 con 0x7f8640060280 2023-11-15T05:08:32.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.723+0000 7f8627fff700 1 -- 172.21.15.154:0/1102098002 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8634010230 con 0x7f8640060280 2023-11-15T05:08:32.724 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:32.725 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:32.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.724+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 msgr2=0x7f8628032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:32.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 0x7f8628032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86300096a0 tx=0x7f8630006b40).stop 2023-11-15T05:08:32.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 msgr2=0x7f86401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:32.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8634013510 tx=0x7f863400b040).stop 2023-11-15T05:08:32.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 shutdown_connections 2023-11-15T05:08:32.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f862802ff60 0x7f8628032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:32.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 --2- 172.21.15.154:0/1102098002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8640060280 0x7f86401066e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:32.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 >> 172.21.15.154:0/1102098002 conn(0x7f86400f0b70 msgr2=0x7f86400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:32.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 shutdown_connections 2023-11-15T05:08:32.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:32.725+0000 7f8645abf700 1 -- 172.21.15.154:0/1102098002 wait complete. 2023-11-15T05:08:32.728 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:32.724172+0000 mon.smithi086 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.154:0/1102098002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:34.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:34.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:34.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:33 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:32.864078+0000 mgr.smithi086.hkfamo (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:35.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:34.864421+0000 mgr.smithi086.hkfamo (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:35.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 -- 172.21.15.154:0/4078516697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 msgr2=0x7f7b200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:35.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 --2- 172.21.15.154:0/4078516697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b200f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f7b10009fb0 tx=0x7f7b10009b30).stop 2023-11-15T05:08:35.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 -- 172.21.15.154:0/4078516697 shutdown_connections 2023-11-15T05:08:35.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 --2- 172.21.15.154:0/4078516697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b200f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:35.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 -- 172.21.15.154:0/4078516697 >> 172.21.15.154:0/4078516697 conn(0x7f7b200f1c70 msgr2=0x7f7b200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:35.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 -- 172.21.15.154:0/4078516697 shutdown_connections 2023-11-15T05:08:35.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.335+0000 7f7b258ed700 1 -- 172.21.15.154:0/4078516697 wait complete. 2023-11-15T05:08:35.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b258ed700 1 Processor -- start 2023-11-15T05:08:35.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b258ed700 1 -- start start 2023-11-15T05:08:35.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b258ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:35.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b258ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20069e20 con 0x7f7b200f6510 2023-11-15T05:08:35.339 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b1effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:35.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b1effd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60638/0 (socket says 172.21.15.154:60638) 2023-11-15T05:08:35.340 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.336+0000 7f7b1effd700 1 -- 172.21.15.154:0/1771292945 learned_addr learned my addr 172.21.15.154:0/1771292945 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:35.351 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b1effd700 1 -- 172.21.15.154:0/1771292945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10005040 con 0x7f7b200f6510 2023-11-15T05:08:35.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b1effd700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f7b10013510 tx=0x7f7b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:35.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b10009320 con 0x7f7b200f6510 2023-11-15T05:08:35.361 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b20069fc0 con 0x7f7b200f6510 2023-11-15T05:08:35.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b10013f00 con 0x7f7b200f6510 2023-11-15T05:08:35.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b10014d90 con 0x7f7b200f6510 2023-11-15T05:08:35.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.337+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2006a450 con 0x7f7b200f6510 2023-11-15T05:08:35.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b100095f0 con 0x7f7b200f6510 2023-11-15T05:08:35.362 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b077fe700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 0x7f7b08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:35.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b100410a0 con 0x7f7b200f6510 2023-11-15T05:08:35.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b20059100 con 0x7f7b200f6510 2023-11-15T05:08:35.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b1e7fc700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 0x7f7b08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:35.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.338+0000 7f7b1e7fc700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 0x7f7b08032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7b140096a0 tx=0x7f7b14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:35.363 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.343+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b1001d7f0 con 0x7f7b200f6510 2023-11-15T05:08:35.548 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.548+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b200fbf60 con 0x7f7b200f6510 2023-11-15T05:08:35.549 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.548+0000 7f7b077fe700 1 -- 172.21.15.154:0/1771292945 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b1002b020 con 0x7f7b200f6510 2023-11-15T05:08:35.549 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:35.550 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:35.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 msgr2=0x7f7b08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:35.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 0x7f7b08032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7b140096a0 tx=0x7f7b14006b40).stop 2023-11-15T05:08:35.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 msgr2=0x7f7b2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:35.551 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f7b10013510 tx=0x7f7b1000b040).stop 2023-11-15T05:08:35.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 shutdown_connections 2023-11-15T05:08:35.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7b0802ff60 0x7f7b08032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:35.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 --2- 172.21.15.154:0/1771292945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b200f6510 0x7f7b2006b9a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:35.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 >> 172.21.15.154:0/1771292945 conn(0x7f7b200f1c70 msgr2=0x7f7b200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:35.552 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 shutdown_connections 2023-11-15T05:08:35.553 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:35.550+0000 7f7b258ed700 1 -- 172.21.15.154:0/1771292945 wait complete. 2023-11-15T05:08:35.553 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:36.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:35.549372+0000 mon.smithi086 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.154:0/1771292945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:36.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:36.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:36.864734+0000 mgr.smithi086.hkfamo (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:38.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 -- 172.21.15.154:0/948053589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 msgr2=0x7fc8780f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:38.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/948053589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f4be0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc86c009fb0 tx=0x7fc86c009b30).stop 2023-11-15T05:08:38.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 -- 172.21.15.154:0/948053589 shutdown_connections 2023-11-15T05:08:38.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/948053589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f4be0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:38.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 -- 172.21.15.154:0/948053589 >> 172.21.15.154:0/948053589 conn(0x7fc8780f02e0 msgr2=0x7fc8780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:38.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 -- 172.21.15.154:0/948053589 shutdown_connections 2023-11-15T05:08:38.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.118+0000 7fc87eaa8700 1 -- 172.21.15.154:0/948053589 wait complete. 2023-11-15T05:08:38.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87eaa8700 1 Processor -- start 2023-11-15T05:08:38.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87eaa8700 1 -- start start 2023-11-15T05:08:38.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87eaa8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:38.123 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87eaa8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8780f57f0 con 0x7fc8780f47c0 2023-11-15T05:08:38.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87c844700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:38.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87c844700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60646/0 (socket says 172.21.15.154:60646) 2023-11-15T05:08:38.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.119+0000 7fc87c844700 1 -- 172.21.15.154:0/1614120481 learned_addr learned my addr 172.21.15.154:0/1614120481 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:38.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc87c844700 1 -- 172.21.15.154:0/1614120481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c005040 con 0x7fc8780f47c0 2023-11-15T05:08:38.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc87c844700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc86c013510 tx=0x7fc86c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:38.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc86c009320 con 0x7fc8780f47c0 2023-11-15T05:08:38.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8780f5990 con 0x7fc8780f47c0 2023-11-15T05:08:38.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc86c013f00 con 0x7fc8780f47c0 2023-11-15T05:08:38.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc86c014d90 con 0x7fc8780f47c0 2023-11-15T05:08:38.127 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.120+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8780f5e20 con 0x7fc8780f47c0 2023-11-15T05:08:38.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc86c0095f0 con 0x7fc8780f47c0 2023-11-15T05:08:38.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc878059100 con 0x7fc8780f47c0 2023-11-15T05:08:38.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc8757fa700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:38.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc86c0410c0 con 0x7fc8780f47c0 2023-11-15T05:08:38.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc877fff700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:38.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.121+0000 7fc877fff700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 0x7fc868032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc8640096a0 tx=0x7fc864006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:38.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.125+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc86c014420 con 0x7fc8780f47c0 2023-11-15T05:08:38.333 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.332+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8780fbf60 con 0x7fc8780f47c0 2023-11-15T05:08:38.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.332+0000 7fc8757fa700 1 -- 172.21.15.154:0/1614120481 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc86c010230 con 0x7fc8780f47c0 2023-11-15T05:08:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:38.335 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:38.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 msgr2=0x7fc868032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:38.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 0x7fc868032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc8640096a0 tx=0x7fc864006b40).stop 2023-11-15T05:08:38.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 msgr2=0x7fc8780f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:38.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc86c013510 tx=0x7fc86c00b040).stop 2023-11-15T05:08:38.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 shutdown_connections 2023-11-15T05:08:38.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc86802ff60 0x7fc868032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:38.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 --2- 172.21.15.154:0/1614120481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8780f47c0 0x7fc8780f7370 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:38.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 >> 172.21.15.154:0/1614120481 conn(0x7fc8780f02e0 msgr2=0x7fc8780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:38.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 shutdown_connections 2023-11-15T05:08:38.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:38.334+0000 7fc87eaa8700 1 -- 172.21.15.154:0/1614120481 wait complete. 2023-11-15T05:08:38.338 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:38.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:38.334632+0000 mon.smithi086 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.154:0/1614120481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:39.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:39.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:39.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:38.865076+0000 mgr.smithi086.hkfamo (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:40.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 -- 172.21.15.154:0/2497135782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 msgr2=0x7fd1c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:40.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/2497135782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fd1b8009fb0 tx=0x7fd1b8009b30).stop 2023-11-15T05:08:40.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 -- 172.21.15.154:0/2497135782 shutdown_connections 2023-11-15T05:08:40.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/2497135782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:40.916 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 -- 172.21.15.154:0/2497135782 >> 172.21.15.154:0/2497135782 conn(0x7fd1c80f0b70 msgr2=0x7fd1c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 -- 172.21.15.154:0/2497135782 shutdown_connections 2023-11-15T05:08:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.912+0000 7fd1ccf05700 1 -- 172.21.15.154:0/2497135782 wait complete. 2023-11-15T05:08:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1ccf05700 1 Processor -- start 2023-11-15T05:08:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1ccf05700 1 -- start start 2023-11-15T05:08:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1ccf05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:40.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1ccf05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80fb820 con 0x7fd1c80f5410 2023-11-15T05:08:40.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1c659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:40.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1c659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:60666/0 (socket says 172.21.15.154:60666) 2023-11-15T05:08:40.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1c659c700 1 -- 172.21.15.154:0/1790183178 learned_addr learned my addr 172.21.15.154:0/1790183178 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:40.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1c659c700 1 -- 172.21.15.154:0/1790183178 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8005040 con 0x7fd1c80f5410 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.913+0000 7fd1c659c700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fd1b8013510 tx=0x7fd1b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.914+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1b8009320 con 0x7fd1c80f5410 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.914+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c80fb9c0 con 0x7fd1c80f5410 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.914+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b8013f00 con 0x7fd1c80f5410 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.914+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1b8014d90 con 0x7fd1c80f5410 2023-11-15T05:08:40.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.914+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c80fbe50 con 0x7fd1c80f5410 2023-11-15T05:08:40.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.915+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c8059100 con 0x7fd1c80f5410 2023-11-15T05:08:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.916+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1b80095f0 con 0x7fd1c80f5410 2023-11-15T05:08:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.917+0000 7fd1b6ffd700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 0x7fd1b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.917+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1b8041240 con 0x7fd1c80f5410 2023-11-15T05:08:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.917+0000 7fd1c5d9b700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 0x7fd1b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.917+0000 7fd1c5d9b700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 0x7fd1b0032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fd1bc0096a0 tx=0x7fd1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:40.922 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:40.921+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1b8014420 con 0x7fd1c80f5410 2023-11-15T05:08:41.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.124+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1c80fae60 con 0x7fd1c80f5410 2023-11-15T05:08:41.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.125+0000 7fd1b6ffd700 1 -- 172.21.15.154:0/1790183178 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1b802b030 con 0x7fd1c80f5410 2023-11-15T05:08:41.127 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:41.127 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:41.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 msgr2=0x7fd1b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:41.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 0x7fd1b0032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fd1bc0096a0 tx=0x7fd1bc006b40).stop 2023-11-15T05:08:41.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 msgr2=0x7fd1c80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:41.129 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fd1b8013510 tx=0x7fd1b800b040).stop 2023-11-15T05:08:41.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 shutdown_connections 2023-11-15T05:08:41.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fd1b002ff60 0x7fd1b0032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:41.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 --2- 172.21.15.154:0/1790183178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1c80f5410 0x7fd1c80fd3a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:41.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 >> 172.21.15.154:0/1790183178 conn(0x7fd1c80f0b70 msgr2=0x7fd1c805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:41.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 shutdown_connections 2023-11-15T05:08:41.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:41.127+0000 7fd1ccf05700 1 -- 172.21.15.154:0/1790183178 wait complete. 2023-11-15T05:08:41.131 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:41.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:40.865431+0000 mgr.smithi086.hkfamo (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:42.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:41 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:41.126917+0000 mon.smithi086 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.154:0/1790183178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:42.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:42.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:43.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:42.865700+0000 mgr.smithi086.hkfamo (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:43.715 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 -- 172.21.15.154:0/3991453334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 msgr2=0x7fa4000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:43.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 --2- 172.21.15.154:0/3991453334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa4000f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fa3f4009fb0 tx=0x7fa3f4009b30).stop 2023-11-15T05:08:43.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 -- 172.21.15.154:0/3991453334 shutdown_connections 2023-11-15T05:08:43.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 --2- 172.21.15.154:0/3991453334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa4000f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:43.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 -- 172.21.15.154:0/3991453334 >> 172.21.15.154:0/3991453334 conn(0x7fa4000f0b70 msgr2=0x7fa4000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:43.716 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 -- 172.21.15.154:0/3991453334 shutdown_connections 2023-11-15T05:08:43.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.713+0000 7fa40613b700 1 -- 172.21.15.154:0/3991453334 wait complete. 2023-11-15T05:08:43.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa40613b700 1 Processor -- start 2023-11-15T05:08:43.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa40613b700 1 -- start start 2023-11-15T05:08:43.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa40613b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:43.717 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa40613b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa400107240 con 0x7fa4000f5410 2023-11-15T05:08:43.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa3ff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:43.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.714+0000 7fa3ff7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37694/0 (socket says 172.21.15.154:37694) 2023-11-15T05:08:43.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3ff7fe700 1 -- 172.21.15.154:0/850931185 learned_addr learned my addr 172.21.15.154:0/850931185 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:43.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3ff7fe700 1 -- 172.21.15.154:0/850931185 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4005040 con 0x7fa4000f5410 2023-11-15T05:08:43.718 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3ff7fe700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fa3f4013510 tx=0x7fa3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:43.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f4009320 con 0x7fa4000f5410 2023-11-15T05:08:43.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa400060220 con 0x7fa4000f5410 2023-11-15T05:08:43.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40010bd40 con 0x7fa4000f5410 2023-11-15T05:08:43.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f4013f00 con 0x7fa4000f5410 2023-11-15T05:08:43.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.715+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f4014d60 con 0x7fa4000f5410 2023-11-15T05:08:43.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.716+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3f4014420 con 0x7fa4000f5410 2023-11-15T05:08:43.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.716+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400045ad0 con 0x7fa4000f5410 2023-11-15T05:08:43.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.716+0000 7fa3e7fff700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 0x7fa3e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:43.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.716+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3f4041360 con 0x7fa4000f5410 2023-11-15T05:08:43.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.717+0000 7fa3feffd700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 0x7fa3e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:43.723 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.717+0000 7fa3feffd700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 0x7fa3e8032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa3f00096a0 tx=0x7fa3f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.721+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3f401c320 con 0x7fa4000f5410 2023-11-15T05:08:43.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.924+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4000fae60 con 0x7fa4000f5410 2023-11-15T05:08:43.925 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.925+0000 7fa3e7fff700 1 -- 172.21.15.154:0/850931185 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3f4012b20 con 0x7fa4000f5410 2023-11-15T05:08:43.926 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:43.926 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:43.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 msgr2=0x7fa3e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:43.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 0x7fa3e8032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa3f00096a0 tx=0x7fa3f0006b40).stop 2023-11-15T05:08:43.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 msgr2=0x7fa40010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:43.927 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fa3f4013510 tx=0x7fa3f400b040).stop 2023-11-15T05:08:43.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 shutdown_connections 2023-11-15T05:08:43.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa3e802ff60 0x7fa3e8032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:43.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 --2- 172.21.15.154:0/850931185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4000f5410 0x7fa40010b850 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:43.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 >> 172.21.15.154:0/850931185 conn(0x7fa4000f0b70 msgr2=0x7fa4000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:43.928 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 shutdown_connections 2023-11-15T05:08:43.929 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:43.926+0000 7fa40613b700 1 -- 172.21.15.154:0/850931185 wait complete. 2023-11-15T05:08:43.929 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:44.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:43.925865+0000 mon.smithi086 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.154:0/850931185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:44.866040+0000 mgr.smithi086.hkfamo (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:45.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:45.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:46.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/3688145449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 msgr2=0x7f7a880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:46.452 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/3688145449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a880f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7a78009fb0 tx=0x7f7a78009b30).stop 2023-11-15T05:08:46.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/3688145449 shutdown_connections 2023-11-15T05:08:46.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/3688145449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a880f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:46.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/3688145449 >> 172.21.15.154:0/3688145449 conn(0x7f7a880f0b70 msgr2=0x7f7a880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:46.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/3688145449 shutdown_connections 2023-11-15T05:08:46.453 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.449+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/3688145449 wait complete. 2023-11-15T05:08:46.454 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8f8c5700 1 Processor -- start 2023-11-15T05:08:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8f8c5700 1 -- start start 2023-11-15T05:08:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8f8c5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8f8c5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88105b20 con 0x7f7a880f5410 2023-11-15T05:08:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8d661700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8d661700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37714/0 (socket says 172.21.15.154:37714) 2023-11-15T05:08:46.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.450+0000 7f7a8d661700 1 -- 172.21.15.154:0/57927351 learned_addr learned my addr 172.21.15.154:0/57927351 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:46.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a8d661700 1 -- 172.21.15.154:0/57927351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a78005040 con 0x7f7a880f5410 2023-11-15T05:08:46.458 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a8d661700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f7a78013510 tx=0x7f7a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:46.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a78009320 con 0x7f7a880f5410 2023-11-15T05:08:46.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a88105d20 con 0x7f7a880f5410 2023-11-15T05:08:46.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a78013f00 con 0x7f7a880f5410 2023-11-15T05:08:46.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a78014d90 con 0x7f7a880f5410 2023-11-15T05:08:46.459 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.451+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a881061b0 con 0x7f7a880f5410 2023-11-15T05:08:46.460 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.452+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a780095f0 con 0x7f7a880f5410 2023-11-15T05:08:46.461 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.452+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a88045ad0 con 0x7f7a880f5410 2023-11-15T05:08:46.461 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.452+0000 7f7a7dffb700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 0x7f7a74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:46.461 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.452+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a78041000 con 0x7f7a880f5410 2023-11-15T05:08:46.462 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.453+0000 7f7a8ce60700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 0x7f7a74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:46.462 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.453+0000 7f7a8ce60700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 0x7f7a74032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7a840096a0 tx=0x7f7a84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:46.462 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.457+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a7801d7f0 con 0x7f7a880f5410 2023-11-15T05:08:46.665 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.664+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a880fae60 con 0x7f7a880f5410 2023-11-15T05:08:46.666 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.665+0000 7f7a7dffb700 1 -- 172.21.15.154:0/57927351 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a78010230 con 0x7f7a880f5410 2023-11-15T05:08:46.666 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:46.666 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:46.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 msgr2=0x7f7a74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:46.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 0x7f7a74032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7a840096a0 tx=0x7f7a84006b40).stop 2023-11-15T05:08:46.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 msgr2=0x7f7a88105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:46.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f7a78013510 tx=0x7f7a7800b040).stop 2023-11-15T05:08:46.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 shutdown_connections 2023-11-15T05:08:46.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f7a7402ff60 0x7f7a74032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:46.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 --2- 172.21.15.154:0/57927351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a880f5410 0x7f7a88105630 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:46.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 >> 172.21.15.154:0/57927351 conn(0x7f7a880f0b70 msgr2=0x7f7a880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:46.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 shutdown_connections 2023-11-15T05:08:46.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:46.667+0000 7f7a8f8c5700 1 -- 172.21.15.154:0/57927351 wait complete. 2023-11-15T05:08:46.670 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:46.666303+0000 mon.smithi086 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.154:0/57927351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:47.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:46.866458+0000 mgr.smithi086.hkfamo (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:48.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:48.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:49.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:48.866763+0000 mgr.smithi086.hkfamo (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:49.302 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 -- 172.21.15.154:0/638541847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 msgr2=0x7fbb38069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:49.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 --2- 172.21.15.154:0/638541847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb38069e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fbb2c009fb0 tx=0x7fbb2c009b30).stop 2023-11-15T05:08:49.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 -- 172.21.15.154:0/638541847 shutdown_connections 2023-11-15T05:08:49.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 --2- 172.21.15.154:0/638541847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb38069e40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:49.303 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 -- 172.21.15.154:0/638541847 >> 172.21.15.154:0/638541847 conn(0x7fbb380f1c70 msgr2=0x7fbb380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:49.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 -- 172.21.15.154:0/638541847 shutdown_connections 2023-11-15T05:08:49.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.301+0000 7fbb3d878700 1 -- 172.21.15.154:0/638541847 wait complete. 2023-11-15T05:08:49.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb3d878700 1 Processor -- start 2023-11-15T05:08:49.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb3d878700 1 -- start start 2023-11-15T05:08:49.304 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb3d878700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:49.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb3d878700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38108de0 con 0x7fbb3806b940 2023-11-15T05:08:49.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb36ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:49.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb36ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37728/0 (socket says 172.21.15.154:37728) 2023-11-15T05:08:49.305 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb36ffd700 1 -- 172.21.15.154:0/557100218 learned_addr learned my addr 172.21.15.154:0/557100218 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:49.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.302+0000 7fbb36ffd700 1 -- 172.21.15.154:0/557100218 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c005040 con 0x7fbb3806b940 2023-11-15T05:08:49.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb36ffd700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbb2c013510 tx=0x7fbb2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:49.307 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2c009320 con 0x7fbb3806b940 2023-11-15T05:08:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38108fe0 con 0x7fbb3806b940 2023-11-15T05:08:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb2c013f00 con 0x7fbb3806b940 2023-11-15T05:08:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2c014dc0 con 0x7fbb3806b940 2023-11-15T05:08:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38109470 con 0x7fbb3806b940 2023-11-15T05:08:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb2c014420 con 0x7fbb3806b940 2023-11-15T05:08:49.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.303+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38045ad0 con 0x7fbb3806b940 2023-11-15T05:08:49.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.304+0000 7fbb1f7fe700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 0x7fbb20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:49.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.304+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb2c041110 con 0x7fbb3806b940 2023-11-15T05:08:49.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.304+0000 7fbb367fc700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 0x7fbb20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:49.309 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.304+0000 7fbb367fc700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 0x7fbb20032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbb280096a0 tx=0x7fbb28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:49.310 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.308+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb2c0095f0 con 0x7fbb3806b940 2023-11-15T05:08:49.514 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.513+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb3810a320 con 0x7fbb3806b940 2023-11-15T05:08:49.515 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.514+0000 7fbb1f7fe700 1 -- 172.21.15.154:0/557100218 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb2c01cb10 con 0x7fbb3806b940 2023-11-15T05:08:49.515 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:49.515 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:49.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 msgr2=0x7fbb20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:49.516 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 0x7fbb20032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbb280096a0 tx=0x7fbb28006b40).stop 2023-11-15T05:08:49.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 msgr2=0x7fbb381088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:49.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbb2c013510 tx=0x7fbb2c00b040).stop 2023-11-15T05:08:49.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 shutdown_connections 2023-11-15T05:08:49.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fbb2002ff60 0x7fbb20032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:49.517 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 --2- 172.21.15.154:0/557100218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3806b940 0x7fbb381088f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:49.518 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 >> 172.21.15.154:0/557100218 conn(0x7fbb380f1c70 msgr2=0x7fbb380f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:49.518 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 shutdown_connections 2023-11-15T05:08:49.518 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:49.515+0000 7fbb3d878700 1 -- 172.21.15.154:0/557100218 wait complete. 2023-11-15T05:08:49.518 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:50.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:49.514871+0000 mon.smithi086 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.154:0/557100218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:50.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:50.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:51.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:50.867182+0000 mgr.smithi086.hkfamo (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:52.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.061+0000 7f583bf6d700 1 -- 172.21.15.154:0/3210741383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 msgr2=0x7f58340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:52.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.061+0000 7f583bf6d700 1 --2- 172.21.15.154:0/3210741383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58340f90f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f582c009fb0 tx=0x7f582c009b30).stop 2023-11-15T05:08:52.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.061+0000 7f583bf6d700 1 -- 172.21.15.154:0/3210741383 shutdown_connections 2023-11-15T05:08:52.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.061+0000 7f583bf6d700 1 --2- 172.21.15.154:0/3210741383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58340f90f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:52.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.061+0000 7f583bf6d700 1 -- 172.21.15.154:0/3210741383 >> 172.21.15.154:0/3210741383 conn(0x7f58340f0b70 msgr2=0x7f58340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:52.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 -- 172.21.15.154:0/3210741383 shutdown_connections 2023-11-15T05:08:52.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 -- 172.21.15.154:0/3210741383 wait complete. 2023-11-15T05:08:52.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 Processor -- start 2023-11-15T05:08:52.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 -- start start 2023-11-15T05:08:52.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:52.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.062+0000 7f583bf6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5834106bd0 con 0x7f5834060280 2023-11-15T05:08:52.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f5839d09700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:52.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f5839d09700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37752/0 (socket says 172.21.15.154:37752) 2023-11-15T05:08:52.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f5839d09700 1 -- 172.21.15.154:0/4074644761 learned_addr learned my addr 172.21.15.154:0/4074644761 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:52.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f5839d09700 1 -- 172.21.15.154:0/4074644761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c005040 con 0x7f5834060280 2023-11-15T05:08:52.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f5839d09700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f582c009830 tx=0x7f582c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:52.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582c009320 con 0x7f5834060280 2023-11-15T05:08:52.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5834106dd0 con 0x7f5834060280 2023-11-15T05:08:52.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f582c0136c0 con 0x7f5834060280 2023-11-15T05:08:52.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582c01ee90 con 0x7f5834060280 2023-11-15T05:08:52.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.063+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5834107260 con 0x7f5834060280 2023-11-15T05:08:52.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.064+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f582c01e420 con 0x7f5834060280 2023-11-15T05:08:52.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.064+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834045ad0 con 0x7f5834060280 2023-11-15T05:08:52.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.064+0000 7f58327fc700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 0x7f581c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:52.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.064+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f582c040ee0 con 0x7f5834060280 2023-11-15T05:08:52.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.065+0000 7f5839508700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 0x7f581c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:52.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.065+0000 7f5839508700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 0x7f581c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f58280096a0 tx=0x7f5828006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:52.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.069+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f582c0095f0 con 0x7f5834060280 2023-11-15T05:08:52.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.277+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5834108110 con 0x7f5834060280 2023-11-15T05:08:52.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.278+0000 7f58327fc700 1 -- 172.21.15.154:0/4074644761 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f582c02c2d0 con 0x7f5834060280 2023-11-15T05:08:52.279 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:52.279 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:52.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.279+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 msgr2=0x7f581c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:52.280 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.279+0000 7f583bf6d700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 0x7f581c0323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f58280096a0 tx=0x7f5828006b40).stop 2023-11-15T05:08:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.279+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 msgr2=0x7f58341066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.279+0000 7f583bf6d700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f582c009830 tx=0x7f582c00b040).stop 2023-11-15T05:08:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 shutdown_connections 2023-11-15T05:08:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f581c02ff10 0x7f581c0323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 --2- 172.21.15.154:0/4074644761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834060280 0x7f58341066e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:52.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 >> 172.21.15.154:0/4074644761 conn(0x7f58340f0b70 msgr2=0x7f58340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:52.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 shutdown_connections 2023-11-15T05:08:52.282 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:52.280+0000 7f583bf6d700 1 -- 172.21.15.154:0/4074644761 wait complete. 2023-11-15T05:08:52.283 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:52.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:52.279024+0000 mon.smithi086 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.154:0/4074644761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:53.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:53.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:53.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:52.867457+0000 mgr.smithi086.hkfamo (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.886+0000 7f7612d15700 1 -- 172.21.15.154:0/1028174634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 msgr2=0x7f760c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.886+0000 7f7612d15700 1 --2- 172.21.15.154:0/1028174634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c0f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f75fc009fb0 tx=0x7f75fc009b30).stop 2023-11-15T05:08:54.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- 172.21.15.154:0/1028174634 shutdown_connections 2023-11-15T05:08:54.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 --2- 172.21.15.154:0/1028174634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c0f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:54.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- 172.21.15.154:0/1028174634 >> 172.21.15.154:0/1028174634 conn(0x7f760c0f09a0 msgr2=0x7f760c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:54.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- 172.21.15.154:0/1028174634 shutdown_connections 2023-11-15T05:08:54.890 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- 172.21.15.154:0/1028174634 wait complete. 2023-11-15T05:08:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 Processor -- start 2023-11-15T05:08:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- start start 2023-11-15T05:08:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.887+0000 7f7612d15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c060240 con 0x7f760c0f51a0 2023-11-15T05:08:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7611d13700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:54.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7611d13700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47594/0 (socket says 172.21.15.154:47594) 2023-11-15T05:08:54.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7611d13700 1 -- 172.21.15.154:0/38345156 learned_addr learned my addr 172.21.15.154:0/38345156 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:54.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7611d13700 1 -- 172.21.15.154:0/38345156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc005040 con 0x7f760c0f51a0 2023-11-15T05:08:54.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7611d13700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f75fc013510 tx=0x7f75fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:54.892 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75fc009320 con 0x7f760c0f51a0 2023-11-15T05:08:54.893 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c060440 con 0x7f760c0f51a0 2023-11-15T05:08:54.893 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75fc013f00 con 0x7f760c0f51a0 2023-11-15T05:08:54.893 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75fc014dc0 con 0x7f760c0f51a0 2023-11-15T05:08:54.893 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.888+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c109860 con 0x7f760c0f51a0 2023-11-15T05:08:54.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.889+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75fc0095f0 con 0x7f760c0f51a0 2023-11-15T05:08:54.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.889+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c045ad0 con 0x7f760c0f51a0 2023-11-15T05:08:54.897 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.889+0000 7f76027fc700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 0x7f75f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:54.898 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.889+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75fc041000 con 0x7f760c0f51a0 2023-11-15T05:08:54.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.889+0000 7f7611512700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 0x7f75f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:54.899 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.890+0000 7f7611512700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 0x7f75f8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f76080096a0 tx=0x7f7608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:54.900 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:54.894+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75fc01c790 con 0x7f760c0f51a0 2023-11-15T05:08:55.098 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.097+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f760c10a080 con 0x7f760c0f51a0 2023-11-15T05:08:55.099 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.098+0000 7f76027fc700 1 -- 172.21.15.154:0/38345156 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75fc010230 con 0x7f760c0f51a0 2023-11-15T05:08:55.099 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:55.099 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:55.100 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.099+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 msgr2=0x7f75f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:55.100 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.099+0000 7f7612d15700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 0x7f75f8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f76080096a0 tx=0x7f7608006b40).stop 2023-11-15T05:08:55.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.099+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 msgr2=0x7f760c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:55.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.099+0000 7f7612d15700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f75fc013510 tx=0x7f75fc00b040).stop 2023-11-15T05:08:55.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 shutdown_connections 2023-11-15T05:08:55.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f75f802ff60 0x7f75f8032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:55.101 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 --2- 172.21.15.154:0/38345156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c0f51a0 0x7f760c1091b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:55.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 >> 172.21.15.154:0/38345156 conn(0x7f760c0f09a0 msgr2=0x7f760c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:55.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 shutdown_connections 2023-11-15T05:08:55.102 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:55.100+0000 7f7612d15700 1 -- 172.21.15.154:0/38345156 wait complete. 2023-11-15T05:08:55.102 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:54.867783+0000 mgr.smithi086.hkfamo (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:56.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:55 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:55.099088+0000 mon.smithi086 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.154:0/38345156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:56.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:56.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:57.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.638+0000 7f4f13e54700 1 -- 172.21.15.154:0/555050219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 msgr2=0x7f4f0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:57.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.638+0000 7f4f13e54700 1 --2- 172.21.15.154:0/555050219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f4f00009fb0 tx=0x7f4f00009b30).stop 2023-11-15T05:08:57.642 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- 172.21.15.154:0/555050219 shutdown_connections 2023-11-15T05:08:57.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 --2- 172.21.15.154:0/555050219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:57.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- 172.21.15.154:0/555050219 >> 172.21.15.154:0/555050219 conn(0x7f4f0c0f0b70 msgr2=0x7f4f0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:57.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- 172.21.15.154:0/555050219 shutdown_connections 2023-11-15T05:08:57.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- 172.21.15.154:0/555050219 wait complete. 2023-11-15T05:08:57.643 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 Processor -- start 2023-11-15T05:08:57.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- start start 2023-11-15T05:08:57.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:57.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.639+0000 7f4f13e54700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c0fb820 con 0x7f4f0c0f5410 2023-11-15T05:08:57.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f11bf0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:57.644 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f11bf0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47612/0 (socket says 172.21.15.154:47612) 2023-11-15T05:08:57.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f11bf0700 1 -- 172.21.15.154:0/12596146 learned_addr learned my addr 172.21.15.154:0/12596146 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:08:57.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f11bf0700 1 -- 172.21.15.154:0/12596146 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f00005040 con 0x7f4f0c0f5410 2023-11-15T05:08:57.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f11bf0700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4f00013510 tx=0x7f4f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:57.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f00009320 con 0x7f4f0c0f5410 2023-11-15T05:08:57.645 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c0fb9c0 con 0x7f4f0c0f5410 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f00013f00 con 0x7f4f0c0f5410 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f00014d90 con 0x7f4f0c0f5410 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.640+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c0fbe50 con 0x7f4f0c0f5410 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.641+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f000095f0 con 0x7f4f0c0f5410 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.641+0000 7f4efe7fc700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 0x7f4ef8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:08:57.646 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.641+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f000410c0 con 0x7f4f0c0f5410 2023-11-15T05:08:57.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.641+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c059100 con 0x7f4f0c0f5410 2023-11-15T05:08:57.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.642+0000 7f4f113ef700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 0x7f4ef8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:08:57.647 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.642+0000 7f4f113ef700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 0x7f4ef8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4f080096a0 tx=0x7f4f08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:08:57.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.646+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f00014420 con 0x7f4f0c0f5410 2023-11-15T05:08:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:56.868079+0000 mgr.smithi086.hkfamo (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:08:57.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.855+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f0c0fae60 con 0x7f4f0c0f5410 2023-11-15T05:08:57.856 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.856+0000 7f4efe7fc700 1 -- 172.21.15.154:0/12596146 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f0002b020 con 0x7f4f0c0f5410 2023-11-15T05:08:57.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:08:57.857 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:08:57.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 msgr2=0x7f4ef8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:57.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 0x7f4ef8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4f080096a0 tx=0x7f4f08006b40).stop 2023-11-15T05:08:57.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 msgr2=0x7f4f0c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:08:57.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4f00013510 tx=0x7f4f0000b040).stop 2023-11-15T05:08:57.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 shutdown_connections 2023-11-15T05:08:57.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4ef802ff60 0x7f4ef8032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:57.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 --2- 172.21.15.154:0/12596146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f0c0f5410 0x7f4f0c0fd3a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:08:57.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 >> 172.21.15.154:0/12596146 conn(0x7f4f0c0f0b70 msgr2=0x7f4f0c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:08:57.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 shutdown_connections 2023-11-15T05:08:57.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:08:57.857+0000 7f4f13e54700 1 -- 172.21.15.154:0/12596146 wait complete. 2023-11-15T05:08:57.860 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:08:58.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:08:57.856846+0000 mon.smithi086 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.154:0/12596146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:08:59.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:08:59.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:08:59.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:08:59 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:08:58.868360+0000 mgr.smithi086.hkfamo (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.482+0000 7f3c6b68c700 1 -- 172.21.15.154:0/911899182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 msgr2=0x7f3c640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.482+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/911899182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c640f90f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3c5c009960 tx=0x7f3c5c004d10).stop 2023-11-15T05:09:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.482+0000 7f3c6b68c700 1 -- 172.21.15.154:0/911899182 shutdown_connections 2023-11-15T05:09:00.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.482+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/911899182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c640f90f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.482+0000 7f3c6b68c700 1 -- 172.21.15.154:0/911899182 >> 172.21.15.154:0/911899182 conn(0x7f3c640f0b70 msgr2=0x7f3c640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 -- 172.21.15.154:0/911899182 shutdown_connections 2023-11-15T05:09:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 -- 172.21.15.154:0/911899182 wait complete. 2023-11-15T05:09:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 Processor -- start 2023-11-15T05:09:00.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 -- start start 2023-11-15T05:09:00.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:00.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.483+0000 7f3c6b68c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64106bb0 con 0x7f3c6405f9c0 2023-11-15T05:09:00.487 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c69428700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c69428700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:47628/0 (socket says 172.21.15.154:47628) 2023-11-15T05:09:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c69428700 1 -- 172.21.15.154:0/1990765525 learned_addr learned my addr 172.21.15.154:0/1990765525 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c69428700 1 -- 172.21.15.154:0/1990765525 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c009640 con 0x7f3c6405f9c0 2023-11-15T05:09:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c69428700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3c5c004b70 tx=0x7f3c5c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c5c018070 con 0x7f3c6405f9c0 2023-11-15T05:09:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c5c00b870 con 0x7f3c6405f9c0 2023-11-15T05:09:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c5c01c560 con 0x7f3c6405f9c0 2023-11-15T05:09:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64106db0 con 0x7f3c6405f9c0 2023-11-15T05:09:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.484+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64107240 con 0x7f3c6405f9c0 2023-11-15T05:09:00.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.485+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c5c013630 con 0x7f3c6405f9c0 2023-11-15T05:09:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.485+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64045ad0 con 0x7f3c6405f9c0 2023-11-15T05:09:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.485+0000 7f3c59ffb700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 0x7f3c50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.485+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c5c041290 con 0x7f3c6405f9c0 2023-11-15T05:09:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.486+0000 7f3c68c27700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 0x7f3c50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.486+0000 7f3c68c27700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 0x7f3c50032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:00.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.490+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c5c004810 con 0x7f3c6405f9c0 2023-11-15T05:09:00.693 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.693+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c641080f0 con 0x7f3c6405f9c0 2023-11-15T05:09:00.694 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.693+0000 7f3c59ffb700 1 -- 172.21.15.154:0/1990765525 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c5c012380 con 0x7f3c6405f9c0 2023-11-15T05:09:00.694 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:00.695 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:00.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 msgr2=0x7f3c50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:00.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 0x7f3c50032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3c540096a0 tx=0x7f3c54006b40).stop 2023-11-15T05:09:00.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 msgr2=0x7f3c641066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:00.696 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3c5c004b70 tx=0x7f3c5c004d10).stop 2023-11-15T05:09:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 shutdown_connections 2023-11-15T05:09:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f3c5002ff60 0x7f3c50032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 --2- 172.21.15.154:0/1990765525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c6405f9c0 0x7f3c641066c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 >> 172.21.15.154:0/1990765525 conn(0x7f3c640f0b70 msgr2=0x7f3c640f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:00.697 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 shutdown_connections 2023-11-15T05:09:00.698 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:00.695+0000 7f3c6b68c700 1 -- 172.21.15.154:0/1990765525 wait complete. 2023-11-15T05:09:00.698 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:00.694402+0000 mon.smithi086 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.154:0/1990765525' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:02.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:02.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:02.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:01 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:00.868668+0000 mgr.smithi086.hkfamo (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:03.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:02.869008+0000 mgr.smithi086.hkfamo (mgr.14162) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:03.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 -- 172.21.15.154:0/1120642589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 msgr2=0x7faf940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:03.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 --2- 172.21.15.154:0/1120642589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf940f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7faf88009fb0 tx=0x7faf88009b30).stop 2023-11-15T05:09:03.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 -- 172.21.15.154:0/1120642589 shutdown_connections 2023-11-15T05:09:03.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 --2- 172.21.15.154:0/1120642589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf940f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:03.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 -- 172.21.15.154:0/1120642589 >> 172.21.15.154:0/1120642589 conn(0x7faf940f1c70 msgr2=0x7faf940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:03.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 -- 172.21.15.154:0/1120642589 shutdown_connections 2023-11-15T05:09:03.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.286+0000 7faf9acbd700 1 -- 172.21.15.154:0/1120642589 wait complete. 2023-11-15T05:09:03.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.287+0000 7faf9acbd700 1 Processor -- start 2023-11-15T05:09:03.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.287+0000 7faf9acbd700 1 -- start start 2023-11-15T05:09:03.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.287+0000 7faf9acbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:03.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.287+0000 7faf9acbd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf94106c20 con 0x7faf940f6510 2023-11-15T05:09:03.293 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.287+0000 7faf98a59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:03.293 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf98a59700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37470/0 (socket says 172.21.15.154:37470) 2023-11-15T05:09:03.293 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf98a59700 1 -- 172.21.15.154:0/2609541944 learned_addr learned my addr 172.21.15.154:0/2609541944 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:03.293 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf98a59700 1 -- 172.21.15.154:0/2609541944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88005040 con 0x7faf940f6510 2023-11-15T05:09:03.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf98a59700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7faf88013510 tx=0x7faf8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:03.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf88009320 con 0x7faf940f6510 2023-11-15T05:09:03.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf88013f00 con 0x7faf940f6510 2023-11-15T05:09:03.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf94106e20 con 0x7faf940f6510 2023-11-15T05:09:03.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf88014d90 con 0x7faf940f6510 2023-11-15T05:09:03.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.288+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf941072b0 con 0x7faf940f6510 2023-11-15T05:09:03.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.289+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf88014420 con 0x7faf940f6510 2023-11-15T05:09:03.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.289+0000 7faf917fa700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 0x7faf84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:03.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.289+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94045ad0 con 0x7faf940f6510 2023-11-15T05:09:03.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.289+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf880410c0 con 0x7faf940f6510 2023-11-15T05:09:03.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.289+0000 7faf93fff700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 0x7faf84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:03.296 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.290+0000 7faf93fff700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 0x7faf84032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:03.297 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.294+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf8806d2b0 con 0x7faf940f6510 2023-11-15T05:09:03.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.499+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf940fc050 con 0x7faf940f6510 2023-11-15T05:09:03.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.500+0000 7faf917fa700 1 -- 172.21.15.154:0/2609541944 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf8801dd50 con 0x7faf940f6510 2023-11-15T05:09:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:03.502 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:03.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 msgr2=0x7faf84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:03.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 0x7faf84032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7faf800096a0 tx=0x7faf80006b40).stop 2023-11-15T05:09:03.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 msgr2=0x7faf94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:03.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7faf88013510 tx=0x7faf8800b040).stop 2023-11-15T05:09:03.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 shutdown_connections 2023-11-15T05:09:03.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7faf8402ff60 0x7faf84032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:03.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 --2- 172.21.15.154:0/2609541944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf940f6510 0x7faf94106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:03.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 >> 172.21.15.154:0/2609541944 conn(0x7faf940f1c70 msgr2=0x7faf940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:03.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 shutdown_connections 2023-11-15T05:09:03.505 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:03.502+0000 7faf9acbd700 1 -- 172.21.15.154:0/2609541944 wait complete. 2023-11-15T05:09:03.505 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:04.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:03.501180+0000 mon.smithi086 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.154:0/2609541944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:04.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:04.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:05.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:04.869446+0000 mgr.smithi086.hkfamo (mgr.14162) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:06.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- 172.21.15.154:0/2537684445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 msgr2=0x7faa000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:06.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 --2- 172.21.15.154:0/2537684445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa000f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa9f4009fb0 tx=0x7fa9f4009b30).stop 2023-11-15T05:09:06.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- 172.21.15.154:0/2537684445 shutdown_connections 2023-11-15T05:09:06.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 --2- 172.21.15.154:0/2537684445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa000f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:06.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- 172.21.15.154:0/2537684445 >> 172.21.15.154:0/2537684445 conn(0x7faa000f1c70 msgr2=0x7faa000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:06.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- 172.21.15.154:0/2537684445 shutdown_connections 2023-11-15T05:09:06.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- 172.21.15.154:0/2537684445 wait complete. 2023-11-15T05:09:06.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 Processor -- start 2023-11-15T05:09:06.070 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- start start 2023-11-15T05:09:06.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:06.071 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.067+0000 7faa06b09700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00106c20 con 0x7faa000f6510 2023-11-15T05:09:06.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7faa048a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:06.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7faa048a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37490/0 (socket says 172.21.15.154:37490) 2023-11-15T05:09:06.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7faa048a5700 1 -- 172.21.15.154:0/820103684 learned_addr learned my addr 172.21.15.154:0/820103684 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:06.073 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7faa048a5700 1 -- 172.21.15.154:0/820103684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4005040 con 0x7faa000f6510 2023-11-15T05:09:06.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7faa048a5700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa9f4013510 tx=0x7fa9f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:06.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.068+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4009320 con 0x7faa000f6510 2023-11-15T05:09:06.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9f4013f00 con 0x7faa000f6510 2023-11-15T05:09:06.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00106e20 con 0x7faa000f6510 2023-11-15T05:09:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4014d90 con 0x7faa000f6510 2023-11-15T05:09:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa001072b0 con 0x7faa000f6510 2023-11-15T05:09:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9f40095f0 con 0x7faa000f6510 2023-11-15T05:09:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa00045ad0 con 0x7faa000f6510 2023-11-15T05:09:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.069+0000 7fa9fd7fa700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 0x7fa9f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:06.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.070+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9f4040fe0 con 0x7faa000f6510 2023-11-15T05:09:06.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.070+0000 7fa9fffff700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 0x7fa9f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:06.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.070+0000 7fa9fffff700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 0x7fa9f0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:06.077 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.074+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9f401d7f0 con 0x7faa000f6510 2023-11-15T05:09:06.281 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.281+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa000fbf60 con 0x7faa000f6510 2023-11-15T05:09:06.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.282+0000 7fa9fd7fa700 1 -- 172.21.15.154:0/820103684 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9f4010230 con 0x7faa000f6510 2023-11-15T05:09:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:06.283 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:06.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 msgr2=0x7fa9f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:06.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 0x7fa9f0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).stop 2023-11-15T05:09:06.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 msgr2=0x7faa00106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:06.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa9f4013510 tx=0x7fa9f400b040).stop 2023-11-15T05:09:06.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 shutdown_connections 2023-11-15T05:09:06.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa9f002ff60 0x7fa9f0032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:06.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 --2- 172.21.15.154:0/820103684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa000f6510 0x7faa00106730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:06.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 >> 172.21.15.154:0/820103684 conn(0x7faa000f1c70 msgr2=0x7faa000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:06.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 shutdown_connections 2023-11-15T05:09:06.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:06.283+0000 7faa06b09700 1 -- 172.21.15.154:0/820103684 wait complete. 2023-11-15T05:09:06.286 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:06.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:06 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:06.282682+0000 mon.smithi086 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.154:0/820103684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:07.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:07.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:06.869787+0000 mgr.smithi086.hkfamo (mgr.14162) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:08.812 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 -- 172.21.15.154:0/2177251179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 msgr2=0x7f08f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:08.813 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/2177251179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f00f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f08e0009fb0 tx=0x7f08e0009b30).stop 2023-11-15T05:09:08.813 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 -- 172.21.15.154:0/2177251179 shutdown_connections 2023-11-15T05:09:08.813 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/2177251179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f00f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:08.813 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 -- 172.21.15.154:0/2177251179 >> 172.21.15.154:0/2177251179 conn(0x7f08f00f0b70 msgr2=0x7f08f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:08.813 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 -- 172.21.15.154:0/2177251179 shutdown_connections 2023-11-15T05:09:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.810+0000 7f08f7dc0700 1 -- 172.21.15.154:0/2177251179 wait complete. 2023-11-15T05:09:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f7dc0700 1 Processor -- start 2023-11-15T05:09:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f7dc0700 1 -- start start 2023-11-15T05:09:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f7dc0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f7dc0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f0106c10 con 0x7f08f00f5410 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f5b5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f5b5c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37500/0 (socket says 172.21.15.154:37500) 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f5b5c700 1 -- 172.21.15.154:0/111148913 learned_addr learned my addr 172.21.15.154:0/111148913 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f5b5c700 1 -- 172.21.15.154:0/111148913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0005040 con 0x7f08f00f5410 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08f5b5c700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f08e0013510 tx=0x7f08e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.811+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e0009320 con 0x7f08f00f5410 2023-11-15T05:09:08.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f0106e10 con 0x7f08f00f5410 2023-11-15T05:09:08.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08e0013f00 con 0x7f08f00f5410 2023-11-15T05:09:08.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e0014d90 con 0x7f08f00f5410 2023-11-15T05:09:08.816 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f01072a0 con 0x7f08f00f5410 2023-11-15T05:09:08.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f0045ad0 con 0x7f08f00f5410 2023-11-15T05:09:08.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.812+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08e00095f0 con 0x7f08f00f5410 2023-11-15T05:09:08.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.813+0000 7f08e67fc700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 0x7f08dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:08.818 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.813+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08e0041000 con 0x7f08f00f5410 2023-11-15T05:09:08.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.813+0000 7f08f535b700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 0x7f08dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:08.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.813+0000 7f08f535b700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 0x7f08dc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f08ec0096a0 tx=0x7f08ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:08.819 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:08.817+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08e0014420 con 0x7f08f00f5410 2023-11-15T05:09:09.025 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.024+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08f0108150 con 0x7f08f00f5410 2023-11-15T05:09:09.026 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.026+0000 7f08e67fc700 1 -- 172.21.15.154:0/111148913 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f08e0010230 con 0x7f08f00f5410 2023-11-15T05:09:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:09.026 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:09.027 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 msgr2=0x7f08dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:09.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 0x7f08dc032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f08ec0096a0 tx=0x7f08ec006b40).stop 2023-11-15T05:09:09.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 msgr2=0x7f08f0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:09.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f08e0013510 tx=0x7f08e000b040).stop 2023-11-15T05:09:09.028 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 shutdown_connections 2023-11-15T05:09:09.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f08dc02ff60 0x7f08dc032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:09.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 --2- 172.21.15.154:0/111148913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08f00f5410 0x7f08f0106720 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:09.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 >> 172.21.15.154:0/111148913 conn(0x7f08f00f0b70 msgr2=0x7f08f005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:09.029 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 shutdown_connections 2023-11-15T05:09:09.030 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:09.027+0000 7f08f7dc0700 1 -- 172.21.15.154:0/111148913 wait complete. 2023-11-15T05:09:09.030 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:09.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:08.870172+0000 mgr.smithi086.hkfamo (mgr.14162) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:10.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:09 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:09.026052+0000 mon.smithi086 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.154:0/111148913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:10.422 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:10.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:11.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:10.870541+0000 mgr.smithi086.hkfamo (mgr.14162) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:11.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 -- 172.21.15.154:0/1955167808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 msgr2=0x7fe9440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:11.631 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 --2- 172.21.15.154:0/1955167808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9440f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fe938009fb0 tx=0x7fe938009b30).stop 2023-11-15T05:09:11.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 -- 172.21.15.154:0/1955167808 shutdown_connections 2023-11-15T05:09:11.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 --2- 172.21.15.154:0/1955167808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9440f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:11.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 -- 172.21.15.154:0/1955167808 >> 172.21.15.154:0/1955167808 conn(0x7fe9440f0b70 msgr2=0x7fe9440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:11.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 -- 172.21.15.154:0/1955167808 shutdown_connections 2023-11-15T05:09:11.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.630+0000 7fe94add5700 1 -- 172.21.15.154:0/1955167808 wait complete. 2023-11-15T05:09:11.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe94add5700 1 Processor -- start 2023-11-15T05:09:11.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe94add5700 1 -- start start 2023-11-15T05:09:11.635 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe94add5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:11.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe94add5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe944106bd0 con 0x7fe944060280 2023-11-15T05:09:11.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe948b71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:11.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe948b71700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:37532/0 (socket says 172.21.15.154:37532) 2023-11-15T05:09:11.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe948b71700 1 -- 172.21.15.154:0/3229071240 learned_addr learned my addr 172.21.15.154:0/3229071240 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:11.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe948b71700 1 -- 172.21.15.154:0/3229071240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe938005040 con 0x7fe944060280 2023-11-15T05:09:11.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.631+0000 7fe948b71700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe938013510 tx=0x7fe93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:11.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe938009320 con 0x7fe944060280 2023-11-15T05:09:11.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe944106dd0 con 0x7fe944060280 2023-11-15T05:09:11.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe938013f00 con 0x7fe944060280 2023-11-15T05:09:11.637 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe938014dc0 con 0x7fe944060280 2023-11-15T05:09:11.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe944107260 con 0x7fe944060280 2023-11-15T05:09:11.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.632+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe938014420 con 0x7fe944060280 2023-11-15T05:09:11.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.633+0000 7fe9417fa700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 0x7fe9340324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:11.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.633+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe938041090 con 0x7fe944060280 2023-11-15T05:09:11.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.633+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe944045ad0 con 0x7fe944060280 2023-11-15T05:09:11.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.633+0000 7fe943fff700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 0x7fe9340324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:11.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.633+0000 7fe943fff700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 0x7fe9340324c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe9300096a0 tx=0x7fe930006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:11.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.637+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9380095f0 con 0x7fe944060280 2023-11-15T05:09:11.848 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.847+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe944107f80 con 0x7fe944060280 2023-11-15T05:09:11.849 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.848+0000 7fe9417fa700 1 -- 172.21.15.154:0/3229071240 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe93801db70 con 0x7fe944060280 2023-11-15T05:09:11.849 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:11.849 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:11.850 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.849+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 msgr2=0x7fe9340324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:11.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.849+0000 7fe94add5700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 0x7fe9340324c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe9300096a0 tx=0x7fe930006b40).stop 2023-11-15T05:09:11.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.849+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 msgr2=0x7fe9441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:11.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.849+0000 7fe94add5700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe938013510 tx=0x7fe93800b040).stop 2023-11-15T05:09:11.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 shutdown_connections 2023-11-15T05:09:11.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fe934030000 0x7fe9340324c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:11.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 --2- 172.21.15.154:0/3229071240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe944060280 0x7fe9441066e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:11.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 >> 172.21.15.154:0/3229071240 conn(0x7fe9440f0b70 msgr2=0x7fe9440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:11.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 shutdown_connections 2023-11-15T05:09:11.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:11.850+0000 7fe94add5700 1 -- 172.21.15.154:0/3229071240 wait complete. 2023-11-15T05:09:11.854 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:12.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:11.848957+0000 mon.smithi086 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.154:0/3229071240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:13.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:13.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:13.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:13 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:12.870963+0000 mgr.smithi086.hkfamo (mgr.14162) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:14.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 -- 172.21.15.154:0/2794337870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 msgr2=0x7f87dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:14.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 --2- 172.21.15.154:0/2794337870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc0f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f87cc009fb0 tx=0x7f87cc009b30).stop 2023-11-15T05:09:14.419 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 -- 172.21.15.154:0/2794337870 shutdown_connections 2023-11-15T05:09:14.419 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 --2- 172.21.15.154:0/2794337870 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc0f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:14.419 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 -- 172.21.15.154:0/2794337870 >> 172.21.15.154:0/2794337870 conn(0x7f87dc0f0b70 msgr2=0x7f87dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:14.420 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 -- 172.21.15.154:0/2794337870 shutdown_connections 2023-11-15T05:09:14.421 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.412+0000 7f87e1249700 1 -- 172.21.15.154:0/2794337870 wait complete. 2023-11-15T05:09:14.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87e1249700 1 Processor -- start 2023-11-15T05:09:14.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87e1249700 1 -- start start 2023-11-15T05:09:14.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87e1249700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:14.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87e1249700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc060240 con 0x7f87dc0f5410 2023-11-15T05:09:14.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87dad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:14.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87dad9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55766/0 (socket says 172.21.15.154:55766) 2023-11-15T05:09:14.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.413+0000 7f87dad9d700 1 -- 172.21.15.154:0/2730470343 learned_addr learned my addr 172.21.15.154:0/2730470343 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:14.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87dad9d700 1 -- 172.21.15.154:0/2730470343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc005040 con 0x7f87dc0f5410 2023-11-15T05:09:14.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87dad9d700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f87cc014000 tx=0x7f87cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:14.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc0095f0 con 0x7f87dc0f5410 2023-11-15T05:09:14.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc060440 con 0x7f87dc0f5410 2023-11-15T05:09:14.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87cc007e70 con 0x7f87dc0f5410 2023-11-15T05:09:14.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87cc014ea0 con 0x7f87dc0f5410 2023-11-15T05:09:14.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.414+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc109ae0 con 0x7f87dc0f5410 2023-11-15T05:09:14.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.415+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc045ad0 con 0x7f87dc0f5410 2023-11-15T05:09:14.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.415+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87cc0079c0 con 0x7f87dc0f5410 2023-11-15T05:09:14.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.415+0000 7f87c37fe700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 0x7f87c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:14.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.415+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87cc0408e0 con 0x7f87dc0f5410 2023-11-15T05:09:14.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.415+0000 7f87da59c700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 0x7f87c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:14.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.416+0000 7f87da59c700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 0x7f87c4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:14.430 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.420+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87cc00f950 con 0x7f87dc0f5410 2023-11-15T05:09:14.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.629+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87dc0fae60 con 0x7f87dc0f5410 2023-11-15T05:09:14.630 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.630+0000 7f87c37fe700 1 -- 172.21.15.154:0/2730470343 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87cc010230 con 0x7f87dc0f5410 2023-11-15T05:09:14.631 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:14.631 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:14.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 msgr2=0x7f87c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:14.632 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 0x7f87c4032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).stop 2023-11-15T05:09:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 msgr2=0x7f87dc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f87cc014000 tx=0x7f87cc00b040).stop 2023-11-15T05:09:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 shutdown_connections 2023-11-15T05:09:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f87c402ff60 0x7f87c4032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 --2- 172.21.15.154:0/2730470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87dc0f5410 0x7f87dc109430 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 >> 172.21.15.154:0/2730470343 conn(0x7f87dc0f0b70 msgr2=0x7f87dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 shutdown_connections 2023-11-15T05:09:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:14.631+0000 7f87e1249700 1 -- 172.21.15.154:0/2730470343 wait complete. 2023-11-15T05:09:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:14.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:14.630638+0000 mon.smithi086 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.154:0/2730470343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:15.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:14.871447+0000 mgr.smithi086.hkfamo (mgr.14162) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:16.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:16.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:17.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 -- 172.21.15.154:0/856113745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 msgr2=0x7f20b40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:17.118 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 --2- 172.21.15.154:0/856113745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b40f90f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f20a8009fb0 tx=0x7f20a8009b30).stop 2023-11-15T05:09:17.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 -- 172.21.15.154:0/856113745 shutdown_connections 2023-11-15T05:09:17.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 --2- 172.21.15.154:0/856113745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b40f90f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:17.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 -- 172.21.15.154:0/856113745 >> 172.21.15.154:0/856113745 conn(0x7f20b40f0b70 msgr2=0x7f20b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:17.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 -- 172.21.15.154:0/856113745 shutdown_connections 2023-11-15T05:09:17.119 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.115+0000 7f20b90e1700 1 -- 172.21.15.154:0/856113745 wait complete. 2023-11-15T05:09:17.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b90e1700 1 Processor -- start 2023-11-15T05:09:17.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b90e1700 1 -- start start 2023-11-15T05:09:17.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b90e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:17.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b90e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4106bd0 con 0x7f20b4060280 2023-11-15T05:09:17.120 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b2d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:17.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b2d9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55786/0 (socket says 172.21.15.154:55786) 2023-11-15T05:09:17.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b2d9d700 1 -- 172.21.15.154:0/796106967 learned_addr learned my addr 172.21.15.154:0/796106967 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:17.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b2d9d700 1 -- 172.21.15.154:0/796106967 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8005040 con 0x7f20b4060280 2023-11-15T05:09:17.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.116+0000 7f20b2d9d700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f20a8013510 tx=0x7f20a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:17.121 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a8009320 con 0x7f20b4060280 2023-11-15T05:09:17.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20a8013f00 con 0x7f20b4060280 2023-11-15T05:09:17.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b4106dd0 con 0x7f20b4060280 2023-11-15T05:09:17.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a8014dc0 con 0x7f20b4060280 2023-11-15T05:09:17.122 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b4107260 con 0x7f20b4060280 2023-11-15T05:09:17.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.117+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20a80095f0 con 0x7f20b4060280 2023-11-15T05:09:17.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.118+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b4045ad0 con 0x7f20b4060280 2023-11-15T05:09:17.124 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.118+0000 7f209b7fe700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:17.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.118+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20a8041080 con 0x7f20b4060280 2023-11-15T05:09:17.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.118+0000 7f20b259c700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:17.125 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.118+0000 7f20b259c700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 0x7f209c032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f20a40096a0 tx=0x7f20a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:17.126 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.123+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20a8014420 con 0x7f20b4060280 2023-11-15T05:09:17.334 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.328+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20b4108110 con 0x7f20b4060280 2023-11-15T05:09:17.335 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.329+0000 7f209b7fe700 1 -- 172.21.15.154:0/796106967 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20a8010230 con 0x7f20b4060280 2023-11-15T05:09:17.335 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:17.335 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:17.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 msgr2=0x7f209c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:17.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 0x7f209c032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f20a40096a0 tx=0x7f20a4006b40).stop 2023-11-15T05:09:17.336 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 msgr2=0x7f20b41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:17.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f20a8013510 tx=0x7f20a800b040).stop 2023-11-15T05:09:17.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 shutdown_connections 2023-11-15T05:09:17.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f209c02ff60 0x7f209c032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:17.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 --2- 172.21.15.154:0/796106967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4060280 0x7f20b41066e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:17.337 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 >> 172.21.15.154:0/796106967 conn(0x7f20b40f0b70 msgr2=0x7f20b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:17.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 shutdown_connections 2023-11-15T05:09:17.338 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:17.330+0000 7f20b90e1700 1 -- 172.21.15.154:0/796106967 wait complete. 2023-11-15T05:09:17.339 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:16.871905+0000 mgr.smithi086.hkfamo (mgr.14162) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:17.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:17.330676+0000 mon.smithi086 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.154:0/796106967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:18.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:18.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:19.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:18.872279+0000 mgr.smithi086.hkfamo (mgr.14162) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:19.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 -- 172.21.15.154:0/1015998073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 msgr2=0x7f03140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:19.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 --2- 172.21.15.154:0/1015998073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f03140f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f0308009fb0 tx=0x7f0308009b30).stop 2023-11-15T05:09:19.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 -- 172.21.15.154:0/1015998073 shutdown_connections 2023-11-15T05:09:19.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 --2- 172.21.15.154:0/1015998073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f03140f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:19.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 -- 172.21.15.154:0/1015998073 >> 172.21.15.154:0/1015998073 conn(0x7f03140f1c70 msgr2=0x7f03140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:19.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 -- 172.21.15.154:0/1015998073 shutdown_connections 2023-11-15T05:09:19.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 -- 172.21.15.154:0/1015998073 wait complete. 2023-11-15T05:09:19.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.859+0000 7f031ac72700 1 Processor -- start 2023-11-15T05:09:19.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f031ac72700 1 -- start start 2023-11-15T05:09:19.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f031ac72700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:19.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f031ac72700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314069e20 con 0x7f03140f6510 2023-11-15T05:09:19.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f0318a0e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:19.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f0318a0e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55804/0 (socket says 172.21.15.154:55804) 2023-11-15T05:09:19.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.860+0000 7f0318a0e700 1 -- 172.21.15.154:0/20316558 learned_addr learned my addr 172.21.15.154:0/20316558 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:19.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f0318a0e700 1 -- 172.21.15.154:0/20316558 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0308005040 con 0x7f03140f6510 2023-11-15T05:09:19.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f0318a0e700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f0308013510 tx=0x7f030800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:19.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0308009320 con 0x7f03140f6510 2023-11-15T05:09:19.870 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0314069fc0 con 0x7f03140f6510 2023-11-15T05:09:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0308013f00 con 0x7f03140f6510 2023-11-15T05:09:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0308014d90 con 0x7f03140f6510 2023-11-15T05:09:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.861+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031406a450 con 0x7f03140f6510 2023-11-15T05:09:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.862+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0308014420 con 0x7f03140f6510 2023-11-15T05:09:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.862+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0314059100 con 0x7f03140f6510 2023-11-15T05:09:19.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.862+0000 7f03117fa700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 0x7f0304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:19.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.862+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0308041050 con 0x7f03140f6510 2023-11-15T05:09:19.872 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.862+0000 7f0313fff700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 0x7f0304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:19.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.867+0000 7f0313fff700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 0x7f0304032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f03000096a0 tx=0x7f0300006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:19.873 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:19.867+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03080095f0 con 0x7f03140f6510 2023-11-15T05:09:20.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.069+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0314045ad0 con 0x7f03140f6510 2023-11-15T05:09:20.072 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.069+0000 7f03117fa700 1 -- 172.21.15.154:0/20316558 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f030801db70 con 0x7f03140f6510 2023-11-15T05:09:20.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:20.073 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:20.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 msgr2=0x7f0304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:20.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 0x7f0304032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f03000096a0 tx=0x7f0300006b40).stop 2023-11-15T05:09:20.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 msgr2=0x7f031406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:20.074 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f0308013510 tx=0x7f030800b040).stop 2023-11-15T05:09:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 shutdown_connections 2023-11-15T05:09:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f030402ff60 0x7f0304032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 --2- 172.21.15.154:0/20316558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03140f6510 0x7f031406b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 >> 172.21.15.154:0/20316558 conn(0x7f03140f1c70 msgr2=0x7f03140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 shutdown_connections 2023-11-15T05:09:20.076 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:20.071+0000 7f031ac72700 1 -- 172.21.15.154:0/20316558 wait complete. 2023-11-15T05:09:20.076 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:20.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:20.071480+0000 mon.smithi086 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.154:0/20316558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:21.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:21.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:21.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:21 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:20.872561+0000 mgr.smithi086.hkfamo (mgr.14162) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:22.590 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 -- 172.21.15.154:0/3723382859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 msgr2=0x7fce980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:22.590 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 --2- 172.21.15.154:0/3723382859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce980f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fce88009fb0 tx=0x7fce88009b30).stop 2023-11-15T05:09:22.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 -- 172.21.15.154:0/3723382859 shutdown_connections 2023-11-15T05:09:22.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 --2- 172.21.15.154:0/3723382859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce980f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:22.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 -- 172.21.15.154:0/3723382859 >> 172.21.15.154:0/3723382859 conn(0x7fce980f1c70 msgr2=0x7fce980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:22.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 -- 172.21.15.154:0/3723382859 shutdown_connections 2023-11-15T05:09:22.591 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.588+0000 7fcea0167700 1 -- 172.21.15.154:0/3723382859 wait complete. 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fcea0167700 1 Processor -- start 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fcea0167700 1 -- start start 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fcea0167700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fcea0167700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce98069e20 con 0x7fce980f6510 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fce9df03700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:22.592 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fce9df03700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:55812/0 (socket says 172.21.15.154:55812) 2023-11-15T05:09:22.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.589+0000 7fce9df03700 1 -- 172.21.15.154:0/1065321844 learned_addr learned my addr 172.21.15.154:0/1065321844 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:22.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fce9df03700 1 -- 172.21.15.154:0/1065321844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce88005040 con 0x7fce980f6510 2023-11-15T05:09:22.593 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fce9df03700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fce88013510 tx=0x7fce8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:22.594 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce88009320 con 0x7fce980f6510 2023-11-15T05:09:22.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98069fc0 con 0x7fce980f6510 2023-11-15T05:09:22.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce88013f00 con 0x7fce980f6510 2023-11-15T05:09:22.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce88014d90 con 0x7fce980f6510 2023-11-15T05:09:22.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.590+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9806a450 con 0x7fce980f6510 2023-11-15T05:09:22.595 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.591+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce880095f0 con 0x7fce980f6510 2023-11-15T05:09:22.596 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.591+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce98059100 con 0x7fce980f6510 2023-11-15T05:09:22.596 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.591+0000 7fce8e7fc700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 0x7fce84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:22.596 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.591+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce88040fe0 con 0x7fce980f6510 2023-11-15T05:09:22.596 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.591+0000 7fce9d702700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 0x7fce84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:22.596 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.592+0000 7fce9d702700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 0x7fce84032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fce940096a0 tx=0x7fce94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:22.597 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.596+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce8806d2b0 con 0x7fce980f6510 2023-11-15T05:09:22.804 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.803+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce980fbf60 con 0x7fce980f6510 2023-11-15T05:09:22.804 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.804+0000 7fce8e7fc700 1 -- 172.21.15.154:0/1065321844 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce88014420 con 0x7fce980f6510 2023-11-15T05:09:22.805 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:22.805 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:22.806 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 msgr2=0x7fce84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:22.806 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 0x7fce84032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fce940096a0 tx=0x7fce94006b40).stop 2023-11-15T05:09:22.806 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 msgr2=0x7fce9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fce88013510 tx=0x7fce8800b040).stop 2023-11-15T05:09:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 shutdown_connections 2023-11-15T05:09:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fce8402ffb0 0x7fce84032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 --2- 172.21.15.154:0/1065321844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce980f6510 0x7fce9806b9a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 >> 172.21.15.154:0/1065321844 conn(0x7fce980f1c70 msgr2=0x7fce980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:22.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 shutdown_connections 2023-11-15T05:09:22.808 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:22.805+0000 7fcea0167700 1 -- 172.21.15.154:0/1065321844 wait complete. 2023-11-15T05:09:22.808 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:23.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:22.804939+0000 mon.smithi086 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.154:0/1065321844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:24.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:22.872967+0000 mgr.smithi086.hkfamo (mgr.14162) 245 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:24.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:24.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:25.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:24.873229+0000 mgr.smithi086.hkfamo (mgr.14162) 246 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:25.484 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 -- 172.21.15.154:0/1065959204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 msgr2=0x7fefb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:25.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 --2- 172.21.15.154:0/1065959204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb40f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fefa4009fb0 tx=0x7fefa4009b30).stop 2023-11-15T05:09:25.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 -- 172.21.15.154:0/1065959204 shutdown_connections 2023-11-15T05:09:25.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 --2- 172.21.15.154:0/1065959204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb40f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:25.485 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 -- 172.21.15.154:0/1065959204 >> 172.21.15.154:0/1065959204 conn(0x7fefb40f0b70 msgr2=0x7fefb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:25.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 -- 172.21.15.154:0/1065959204 shutdown_connections 2023-11-15T05:09:25.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.483+0000 7fefba3cc700 1 -- 172.21.15.154:0/1065959204 wait complete. 2023-11-15T05:09:25.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.484+0000 7fefba3cc700 1 Processor -- start 2023-11-15T05:09:25.486 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.484+0000 7fefba3cc700 1 -- start start 2023-11-15T05:09:25.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.484+0000 7fefba3cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:25.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.484+0000 7fefba3cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb4060220 con 0x7fefb40f5410 2023-11-15T05:09:25.488 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:25.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb3fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45166/0 (socket says 172.21.15.154:45166) 2023-11-15T05:09:25.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb3fff700 1 -- 172.21.15.154:0/3260664416 learned_addr learned my addr 172.21.15.154:0/3260664416 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:25.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb3fff700 1 -- 172.21.15.154:0/3260664416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa4005040 con 0x7fefb40f5410 2023-11-15T05:09:25.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb3fff700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fefa4013510 tx=0x7fefa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:25.489 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa4009320 con 0x7fefb40f5410 2023-11-15T05:09:25.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb4060420 con 0x7fefb40f5410 2023-11-15T05:09:25.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefa4013f00 con 0x7fefb40f5410 2023-11-15T05:09:25.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.485+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa4014d90 con 0x7fefb40f5410 2023-11-15T05:09:25.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.486+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb4109b00 con 0x7fefb40f5410 2023-11-15T05:09:25.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.486+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefa40095f0 con 0x7fefb40f5410 2023-11-15T05:09:25.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.486+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb4059100 con 0x7fefb40f5410 2023-11-15T05:09:25.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.486+0000 7fefb0ff9700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 0x7fef9c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:25.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.486+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefa40410a0 con 0x7fefb40f5410 2023-11-15T05:09:25.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.487+0000 7fefb37fe700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 0x7fef9c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:25.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.487+0000 7fefb37fe700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 0x7fef9c032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fefa80096a0 tx=0x7fefa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:25.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.491+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefa406d2b0 con 0x7fefb40f5410 2023-11-15T05:09:25.699 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.699+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefb4045ad0 con 0x7fefb40f5410 2023-11-15T05:09:25.700 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.699+0000 7fefb0ff9700 1 -- 172.21.15.154:0/3260664416 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefa4014420 con 0x7fefb40f5410 2023-11-15T05:09:25.701 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:25.701 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:25.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 msgr2=0x7fef9c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:25.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 0x7fef9c032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fefa80096a0 tx=0x7fefa8006b40).stop 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 msgr2=0x7fefb4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fefa4013510 tx=0x7fefa400b040).stop 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 shutdown_connections 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fef9c02ffb0 0x7fef9c032470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 --2- 172.21.15.154:0/3260664416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefb40f5410 0x7fefb4109450 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:25.703 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 >> 172.21.15.154:0/3260664416 conn(0x7fefb40f0b70 msgr2=0x7fefb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:25.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 shutdown_connections 2023-11-15T05:09:25.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:25.701+0000 7fefba3cc700 1 -- 172.21.15.154:0/3260664416 wait complete. 2023-11-15T05:09:25.704 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:26.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:24.933858+0000 mon.smithi086 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:09:26.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:24.939991+0000 mon.smithi086 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.86:0/582051164' entity='mgr.smithi086.hkfamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.hkfamo/trash_purge_schedule"}]: dispatch 2023-11-15T05:09:26.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:25 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:25.700598+0000 mon.smithi086 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.154:0/3260664416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:27.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:27.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:27.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:27 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:26.873508+0000 mgr.smithi086.hkfamo (mgr.14162) 247 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:28.283 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 -- 172.21.15.154:0/1315467444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 msgr2=0x7fc8100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:28.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1315467444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc8100f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fc7f8009fb0 tx=0x7fc7f8009b30).stop 2023-11-15T05:09:28.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 -- 172.21.15.154:0/1315467444 shutdown_connections 2023-11-15T05:09:28.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1315467444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc8100f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:28.284 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 -- 172.21.15.154:0/1315467444 >> 172.21.15.154:0/1315467444 conn(0x7fc8100f1c70 msgr2=0x7fc8100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:28.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 -- 172.21.15.154:0/1315467444 shutdown_connections 2023-11-15T05:09:28.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.282+0000 7fc814f7a700 1 -- 172.21.15.154:0/1315467444 wait complete. 2023-11-15T05:09:28.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc814f7a700 1 Processor -- start 2023-11-15T05:09:28.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc814f7a700 1 -- start start 2023-11-15T05:09:28.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc814f7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:28.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc814f7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc810106c20 con 0x7fc8100f6510 2023-11-15T05:09:28.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc80e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:28.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc80e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45172/0 (socket says 172.21.15.154:45172) 2023-11-15T05:09:28.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.283+0000 7fc80e59c700 1 -- 172.21.15.154:0/1823050398 learned_addr learned my addr 172.21.15.154:0/1823050398 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:28.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc80e59c700 1 -- 172.21.15.154:0/1823050398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8005040 con 0x7fc8100f6510 2023-11-15T05:09:28.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc80e59c700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:28.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8009320 con 0x7fc8100f6510 2023-11-15T05:09:28.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7f8013f00 con 0x7fc8100f6510 2023-11-15T05:09:28.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc810106e20 con 0x7fc8100f6510 2023-11-15T05:09:28.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8014d90 con 0x7fc8100f6510 2023-11-15T05:09:28.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.284+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8101072b0 con 0x7fc8100f6510 2023-11-15T05:09:28.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.285+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7f80095f0 con 0x7fc8100f6510 2023-11-15T05:09:28.289 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.285+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc810045ad0 con 0x7fc8100f6510 2023-11-15T05:09:28.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.285+0000 7fc806ffd700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 0x7fc7fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:28.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.285+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7f8040fd0 con 0x7fc8100f6510 2023-11-15T05:09:28.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.285+0000 7fc80dd9b700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 0x7fc7fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:28.290 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.286+0000 7fc80dd9b700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 0x7fc7fc032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc8000096a0 tx=0x7fc800006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:28.291 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.290+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7f8014420 con 0x7fc8100f6510 2023-11-15T05:09:28.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.496+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8100fbf60 con 0x7fc8100f6510 2023-11-15T05:09:28.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.496+0000 7fc806ffd700 1 -- 172.21.15.154:0/1823050398 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7f801d7f0 con 0x7fc8100f6510 2023-11-15T05:09:28.498 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:28.498 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:28.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 msgr2=0x7fc7fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:28.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 0x7fc7fc032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc8000096a0 tx=0x7fc800006b40).stop 2023-11-15T05:09:28.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 msgr2=0x7fc810106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:28.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).stop 2023-11-15T05:09:28.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 shutdown_connections 2023-11-15T05:09:28.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7fc02ffb0 0x7fc7fc032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:28.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 --2- 172.21.15.154:0/1823050398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8100f6510 0x7fc810106730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:28.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 >> 172.21.15.154:0/1823050398 conn(0x7fc8100f1c70 msgr2=0x7fc8100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:28.500 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 shutdown_connections 2023-11-15T05:09:28.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:28.498+0000 7fc814f7a700 1 -- 172.21.15.154:0/1823050398 wait complete. 2023-11-15T05:09:28.501 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:28.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:28 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:28.497624+0000 mon.smithi086 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.154:0/1823050398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:29.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:29.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:29.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:29 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:28.873831+0000 mgr.smithi086.hkfamo (mgr.14162) 248 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:31.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/1707939929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 msgr2=0x7f0eb00f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:31.041 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/1707939929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb00f66e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f0eac009fb0 tx=0x7f0eac009b30).stop 2023-11-15T05:09:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/1707939929 shutdown_connections 2023-11-15T05:09:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/1707939929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb00f66e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/1707939929 >> 172.21.15.154:0/1707939929 conn(0x7f0eb00f1ac0 msgr2=0x7f0eb00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/1707939929 shutdown_connections 2023-11-15T05:09:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.039+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/1707939929 wait complete. 2023-11-15T05:09:31.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb6bfc700 1 Processor -- start 2023-11-15T05:09:31.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb6bfc700 1 -- start start 2023-11-15T05:09:31.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb6bfc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:31.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb6bfc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0106a10 con 0x7f0eb00f62c0 2023-11-15T05:09:31.043 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb5bfa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:31.044 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb5bfa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:45190/0 (socket says 172.21.15.154:45190) 2023-11-15T05:09:31.044 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.040+0000 7f0eb5bfa700 1 -- 172.21.15.154:0/3261894495 learned_addr learned my addr 172.21.15.154:0/3261894495 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:31.044 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0eb5bfa700 1 -- 172.21.15.154:0/3261894495 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac005040 con 0x7f0eb00f62c0 2023-11-15T05:09:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0eb5bfa700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f0eac013510 tx=0x7f0eac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eac009320 con 0x7f0eb00f62c0 2023-11-15T05:09:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb0106c10 con 0x7f0eb00f62c0 2023-11-15T05:09:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0eac013f00 con 0x7f0eb00f62c0 2023-11-15T05:09:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eac014dc0 con 0x7f0eb00f62c0 2023-11-15T05:09:31.047 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.041+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb01070a0 con 0x7f0eb00f62c0 2023-11-15T05:09:31.047 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.042+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0eac0095f0 con 0x7f0eb00f62c0 2023-11-15T05:09:31.047 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.042+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb0045ad0 con 0x7f0eb00f62c0 2023-11-15T05:09:31.047 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.042+0000 7f0ea67fc700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 0x7f0e9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:31.048 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.042+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0eac0410c0 con 0x7f0eb00f62c0 2023-11-15T05:09:31.048 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.042+0000 7f0eb53f9700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 0x7f0e9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:31.048 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.043+0000 7f0eb53f9700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 0x7f0e9c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0ea00096a0 tx=0x7f0ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:31.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.047+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0eac01c790 con 0x7f0eb00f62c0 2023-11-15T05:09:31.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:30 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:30.874153+0000 mgr.smithi086.hkfamo (mgr.14162) 249 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:31.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.253+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eb0107f50 con 0x7f0eb00f62c0 2023-11-15T05:09:31.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.254+0000 7f0ea67fc700 1 -- 172.21.15.154:0/3261894495 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0eac010230 con 0x7f0eb00f62c0 2023-11-15T05:09:31.254 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:31.255 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:31.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 msgr2=0x7f0e9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:31.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 0x7f0e9c032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0ea00096a0 tx=0x7f0ea0006b40).stop 2023-11-15T05:09:31.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 msgr2=0x7f0eb0106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:31.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f0eac013510 tx=0x7f0eac00b040).stop 2023-11-15T05:09:31.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 shutdown_connections 2023-11-15T05:09:31.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0e9c02ff60 0x7f0e9c032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:31.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 --2- 172.21.15.154:0/3261894495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0eb00f62c0 0x7f0eb0106520 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:31.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 >> 172.21.15.154:0/3261894495 conn(0x7f0eb00f1ac0 msgr2=0x7f0eb00f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:31.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 shutdown_connections 2023-11-15T05:09:31.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:31.255+0000 7f0eb6bfc700 1 -- 172.21.15.154:0/3261894495 wait complete. 2023-11-15T05:09:31.258 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:32.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:31 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:31.254817+0000 mon.smithi086 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.154:0/3261894495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:32.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:32.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:33.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:32 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:32.874571+0000 mgr.smithi086.hkfamo (mgr.14162) 250 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:33.836 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 -- 172.21.15.154:0/2477401097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 msgr2=0x7f0f000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:33.836 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 --2- 172.21.15.154:0/2477401097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f000f66c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f0efc009fb0 tx=0x7f0efc009b30).stop 2023-11-15T05:09:33.836 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 -- 172.21.15.154:0/2477401097 shutdown_connections 2023-11-15T05:09:33.836 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 --2- 172.21.15.154:0/2477401097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f000f66c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:33.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 -- 172.21.15.154:0/2477401097 >> 172.21.15.154:0/2477401097 conn(0x7f0f000f1aa0 msgr2=0x7f0f000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:33.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 -- 172.21.15.154:0/2477401097 shutdown_connections 2023-11-15T05:09:33.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 -- 172.21.15.154:0/2477401097 wait complete. 2023-11-15T05:09:33.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.830+0000 7f0f0677b700 1 Processor -- start 2023-11-15T05:09:33.837 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f0677b700 1 -- start start 2023-11-15T05:09:33.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f0677b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:33.838 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f0677b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001069a0 con 0x7f0f000f62a0 2023-11-15T05:09:33.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f05779700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:33.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f05779700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51834/0 (socket says 172.21.15.154:51834) 2023-11-15T05:09:33.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f05779700 1 -- 172.21.15.154:0/3851201026 learned_addr learned my addr 172.21.15.154:0/3851201026 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:33.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f05779700 1 -- 172.21.15.154:0/3851201026 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc005040 con 0x7f0f000f62a0 2023-11-15T05:09:33.839 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f05779700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0efc013510 tx=0x7f0efc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:33.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0efc009320 con 0x7f0f000f62a0 2023-11-15T05:09:33.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f00106ba0 con 0x7f0f000f62a0 2023-11-15T05:09:33.840 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0efc013f00 con 0x7f0f000f62a0 2023-11-15T05:09:33.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.831+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0efc014dc0 con 0x7f0f000f62a0 2023-11-15T05:09:33.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.832+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f00107030 con 0x7f0f000f62a0 2023-11-15T05:09:33.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.832+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0efc01d3d0 con 0x7f0f000f62a0 2023-11-15T05:09:33.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.833+0000 7f0ef5ffb700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 0x7f0eec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:33.841 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.833+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f00045ad0 con 0x7f0f000f62a0 2023-11-15T05:09:33.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.833+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0efc0413b0 con 0x7f0f000f62a0 2023-11-15T05:09:33.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.833+0000 7f0f04f78700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 0x7f0eec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:33.842 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.833+0000 7f0f04f78700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 0x7f0eec032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0ef00096a0 tx=0x7f0ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:33.843 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:33.837+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0efc01cb10 con 0x7f0f000f62a0 2023-11-15T05:09:34.046 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.045+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f00107e80 con 0x7f0f000f62a0 2023-11-15T05:09:34.047 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.046+0000 7f0ef5ffb700 1 -- 172.21.15.154:0/3851201026 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0efc01cb10 con 0x7f0f000f62a0 2023-11-15T05:09:34.047 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:34.047 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 msgr2=0x7f0eec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 0x7f0eec032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0ef00096a0 tx=0x7f0ef0006b40).stop 2023-11-15T05:09:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 msgr2=0x7f0f001064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f0efc013510 tx=0x7f0efc00b040).stop 2023-11-15T05:09:34.049 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 shutdown_connections 2023-11-15T05:09:34.050 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f0eec02ff60 0x7f0eec032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:34.050 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 --2- 172.21.15.154:0/3851201026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f000f62a0 0x7f0f001064b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:34.050 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 >> 172.21.15.154:0/3851201026 conn(0x7f0f000f1aa0 msgr2=0x7f0f000f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:34.050 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 shutdown_connections 2023-11-15T05:09:34.050 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:34.048+0000 7f0f0677b700 1 -- 172.21.15.154:0/3851201026 wait complete. 2023-11-15T05:09:34.051 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:34.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:34 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:34.047101+0000 mon.smithi086 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.154:0/3851201026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:35.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:35.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:35.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:35 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:34.874845+0000 mgr.smithi086.hkfamo (mgr.14162) 251 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:36.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 -- 172.21.15.154:0/3104127554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 msgr2=0x7f92c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:36.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 --2- 172.21.15.154:0/3104127554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c00f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f92b0009fb0 tx=0x7f92b0009b30).stop 2023-11-15T05:09:36.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 -- 172.21.15.154:0/3104127554 shutdown_connections 2023-11-15T05:09:36.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 --2- 172.21.15.154:0/3104127554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c00f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:36.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 -- 172.21.15.154:0/3104127554 >> 172.21.15.154:0/3104127554 conn(0x7f92c00f0b70 msgr2=0x7f92c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:36.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 -- 172.21.15.154:0/3104127554 shutdown_connections 2023-11-15T05:09:36.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.668+0000 7f92c60cb700 1 -- 172.21.15.154:0/3104127554 wait complete. 2023-11-15T05:09:36.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.669+0000 7f92c60cb700 1 Processor -- start 2023-11-15T05:09:36.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.669+0000 7f92c60cb700 1 -- start start 2023-11-15T05:09:36.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.669+0000 7f92c60cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:36.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.669+0000 7f92c60cb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0060240 con 0x7f92c00f5410 2023-11-15T05:09:36.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92bf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:36.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92bf7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51848/0 (socket says 172.21.15.154:51848) 2023-11-15T05:09:36.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92bf7fe700 1 -- 172.21.15.154:0/88086502 learned_addr learned my addr 172.21.15.154:0/88086502 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:36.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92bf7fe700 1 -- 172.21.15.154:0/88086502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b0005040 con 0x7f92c00f5410 2023-11-15T05:09:36.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92bf7fe700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f92b0014000 tx=0x7f92b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:36.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b00095f0 con 0x7f92c00f5410 2023-11-15T05:09:36.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c0060440 con 0x7f92c00f5410 2023-11-15T05:09:36.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92b0007e70 con 0x7f92c00f5410 2023-11-15T05:09:36.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b0014ea0 con 0x7f92c00f5410 2023-11-15T05:09:36.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.670+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c0109ae0 con 0x7f92c00f5410 2023-11-15T05:09:36.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.671+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92b001d4d0 con 0x7f92c00f5410 2023-11-15T05:09:36.676 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.671+0000 7f92a7fff700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 0x7f92a80324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:36.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.671+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c0045ad0 con 0x7f92c00f5410 2023-11-15T05:09:36.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.671+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92b0040950 con 0x7f92c00f5410 2023-11-15T05:09:36.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.672+0000 7f92beffd700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 0x7f92a80324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:36.677 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.672+0000 7f92beffd700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 0x7f92a80324c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f92b40096a0 tx=0x7f92b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:36.678 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.676+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92b000f710 con 0x7f92c00f5410 2023-11-15T05:09:36.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.882+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92c00faf50 con 0x7f92c00f5410 2023-11-15T05:09:36.883 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.883+0000 7f92a7fff700 1 -- 172.21.15.154:0/88086502 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92b00252f0 con 0x7f92c00f5410 2023-11-15T05:09:36.884 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:36.884 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:36.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 msgr2=0x7f92a80324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:36.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 0x7f92a80324c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f92b40096a0 tx=0x7f92b4006b40).stop 2023-11-15T05:09:36.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 msgr2=0x7f92c0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:36.886 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f92b0014000 tx=0x7f92b000b040).stop 2023-11-15T05:09:36.886 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 shutdown_connections 2023-11-15T05:09:36.886 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f92a8030000 0x7f92a80324c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:36.886 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 --2- 172.21.15.154:0/88086502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92c00f5410 0x7f92c0109430 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 >> 172.21.15.154:0/88086502 conn(0x7f92c00f0b70 msgr2=0x7f92c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 shutdown_connections 2023-11-15T05:09:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:36.884+0000 7f92c60cb700 1 -- 172.21.15.154:0/88086502 wait complete. 2023-11-15T05:09:36.888 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:36.875144+0000 mgr.smithi086.hkfamo (mgr.14162) 252 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:37.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:37 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:36.883906+0000 mon.smithi086 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.154:0/88086502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:38.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:38.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:38.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:09:37.276887+0000 mgr.smithi086.hkfamo (mgr.14162) 253 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi086 ... 2023-11-15T05:09:38.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Verifying port 9100 ... 2023-11-15T05:09:38.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 1 from systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086 2023-11-15T05:09:38.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: systemctl: stderr Job for ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086.service failed because a timeout was exceeded. 2023-11-15T05:09:38.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: systemctl: stderr See "systemctl status ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086.service" and "journalctl -xe" for details. 2023-11-15T05:09:38.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:09:38.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:09:38.741 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: main() 2023-11-15T05:09:38.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:09:38.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: r = ctx.func(ctx) 2023-11-15T05:09:38.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:09:38.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: return func(ctx) 2023-11-15T05:09:38.742 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T05:09:38.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ports=daemon_ports) 2023-11-15T05:09:38.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T05:09:38.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T05:09:38.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T05:09:38.743 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T05:09:38.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T05:09:38.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T05:09:38.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: RuntimeError: Failed command: systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086 2023-11-15T05:09:38.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:09:38.744 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T05:09:38.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: yield (conn, connr) 2023-11-15T05:09:38.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T05:09:38.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: code, '\n'.join(err))) 2023-11-15T05:09:38.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi086 ... 2023-11-15T05:09:38.745 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Verifying port 9100 ... 2023-11-15T05:09:38.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Non-zero exit code 1 from systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086 2023-11-15T05:09:38.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: systemctl: stderr Job for ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086.service failed because a timeout was exceeded. 2023-11-15T05:09:38.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: systemctl: stderr See "systemctl status ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086.service" and "journalctl -xe" for details. 2023-11-15T05:09:38.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: Traceback (most recent call last): 2023-11-15T05:09:38.746 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: main() 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: r = ctx.func(ctx) 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: return func(ctx) 2023-11-15T05:09:38.747 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T05:09:38.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: ports=daemon_ports) 2023-11-15T05:09:38.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T05:09:38.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T05:09:38.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T05:09:38.748 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T05:09:38.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: File "/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T05:09:38.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T05:09:38.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: RuntimeError: Failed command: systemctl start ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@node-exporter.smithi086 2023-11-15T05:09:38.750 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:38 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cephadm 2023-11-15T05:09:37.282216+0000 mgr.smithi086.hkfamo (mgr.14162) 254 : cephadm [INF] Deploying daemon prometheus.smithi086 on smithi086 2023-11-15T05:09:39.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.430+0000 7f5c936cc700 1 -- 172.21.15.154:0/3313487210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 msgr2=0x7f5c8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:39.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.430+0000 7f5c936cc700 1 --2- 172.21.15.154:0/3313487210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c0f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f5c7c009fb0 tx=0x7f5c7c009b30).stop 2023-11-15T05:09:39.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.430+0000 7f5c936cc700 1 -- 172.21.15.154:0/3313487210 shutdown_connections 2023-11-15T05:09:39.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.430+0000 7f5c936cc700 1 --2- 172.21.15.154:0/3313487210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c0f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:39.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 -- 172.21.15.154:0/3313487210 >> 172.21.15.154:0/3313487210 conn(0x7f5c8c0f0b70 msgr2=0x7f5c8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 -- 172.21.15.154:0/3313487210 shutdown_connections 2023-11-15T05:09:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 -- 172.21.15.154:0/3313487210 wait complete. 2023-11-15T05:09:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 Processor -- start 2023-11-15T05:09:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 -- start start 2023-11-15T05:09:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.431+0000 7f5c936cc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:39.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c936cc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c105b20 con 0x7f5c8c0f5410 2023-11-15T05:09:39.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c91468700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:39.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c91468700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51870/0 (socket says 172.21.15.154:51870) 2023-11-15T05:09:39.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c91468700 1 -- 172.21.15.154:0/1838952841 learned_addr learned my addr 172.21.15.154:0/1838952841 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c91468700 1 -- 172.21.15.154:0/1838952841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c005040 con 0x7f5c8c0f5410 2023-11-15T05:09:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c91468700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5c7c013510 tx=0x7f5c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c7c009320 con 0x7f5c8c0f5410 2023-11-15T05:09:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.432+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c105d20 con 0x7f5c8c0f5410 2023-11-15T05:09:39.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.433+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c7c013f00 con 0x7f5c8c0f5410 2023-11-15T05:09:39.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.433+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c7c014d90 con 0x7f5c8c0f5410 2023-11-15T05:09:39.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.433+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c1061b0 con 0x7f5c8c0f5410 2023-11-15T05:09:39.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.433+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c7c0095f0 con 0x7f5c8c0f5410 2023-11-15T05:09:39.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.434+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c045ad0 con 0x7f5c8c0f5410 2023-11-15T05:09:39.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.434+0000 7f5c81ffb700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 0x7f5c78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.434+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c7c0410c0 con 0x7f5c8c0f5410 2023-11-15T05:09:39.439 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.434+0000 7f5c90c67700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 0x7f5c78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:39.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.434+0000 7f5c90c67700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 0x7f5c78032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f5c880096a0 tx=0x7f5c88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:39.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.438+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c7c01d7f0 con 0x7f5c8c0f5410 2023-11-15T05:09:39.648 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.647+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c8c0fae60 con 0x7f5c8c0f5410 2023-11-15T05:09:39.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.648+0000 7f5c81ffb700 1 -- 172.21.15.154:0/1838952841 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c7c010230 con 0x7f5c8c0f5410 2023-11-15T05:09:39.649 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:39.649 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:39.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 msgr2=0x7f5c78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:39.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 0x7f5c78032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f5c880096a0 tx=0x7f5c88006b40).stop 2023-11-15T05:09:39.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 msgr2=0x7f5c8c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:39.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f5c7c013510 tx=0x7f5c7c00b040).stop 2023-11-15T05:09:39.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 shutdown_connections 2023-11-15T05:09:39.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f5c7802ff60 0x7f5c78032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:39.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 --2- 172.21.15.154:0/1838952841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c8c0f5410 0x7f5c8c105630 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:39.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 >> 172.21.15.154:0/1838952841 conn(0x7f5c8c0f0b70 msgr2=0x7f5c8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:39.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 shutdown_connections 2023-11-15T05:09:39.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:39.650+0000 7f5c936cc700 1 -- 172.21.15.154:0/1838952841 wait complete. 2023-11-15T05:09:39.653 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:39.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:39 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:38.875525+0000 mgr.smithi086.hkfamo (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:40.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:40 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:39.649310+0000 mon.smithi086 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.154:0/1838952841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:41.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:41.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:41.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:41 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:40.875954+0000 mgr.smithi086.hkfamo (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 -- 172.21.15.154:0/1429682133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 msgr2=0x7f54340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:42.253 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 --2- 172.21.15.154:0/1429682133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f5424009fb0 tx=0x7f5424009b30).stop 2023-11-15T05:09:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 -- 172.21.15.154:0/1429682133 shutdown_connections 2023-11-15T05:09:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 --2- 172.21.15.154:0/1429682133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 -- 172.21.15.154:0/1429682133 >> 172.21.15.154:0/1429682133 conn(0x7f54340f0b70 msgr2=0x7f54340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 -- 172.21.15.154:0/1429682133 shutdown_connections 2023-11-15T05:09:42.254 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 -- 172.21.15.154:0/1429682133 wait complete. 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.249+0000 7f543bc17700 1 Processor -- start 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f543bc17700 1 -- start start 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f543bc17700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f543bc17700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54340fb820 con 0x7f54340f5410 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f54399b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f54399b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:51892/0 (socket says 172.21.15.154:51892) 2023-11-15T05:09:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f54399b3700 1 -- 172.21.15.154:0/3023372468 learned_addr learned my addr 172.21.15.154:0/3023372468 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f54399b3700 1 -- 172.21.15.154:0/3023372468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424005040 con 0x7f54340f5410 2023-11-15T05:09:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f54399b3700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f5424014000 tx=0x7f542400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.250+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54240095f0 con 0x7f54340f5410 2023-11-15T05:09:42.256 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54340fb9c0 con 0x7f54340f5410 2023-11-15T05:09:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5424007e70 con 0x7f54340f5410 2023-11-15T05:09:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5424014ea0 con 0x7f54340f5410 2023-11-15T05:09:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54340fbe50 con 0x7f54340f5410 2023-11-15T05:09:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5424025070 con 0x7f54340f5410 2023-11-15T05:09:42.257 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5434059100 con 0x7f54340f5410 2023-11-15T05:09:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.251+0000 7f542a7fc700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.252+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5424040970 con 0x7f54340f5410 2023-11-15T05:09:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.252+0000 7f54391b2700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:42.258 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.252+0000 7f54391b2700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f54300096a0 tx=0x7f5430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:42.259 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.256+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f542400f300 con 0x7f54340f5410 2023-11-15T05:09:42.464 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.463+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54340fae60 con 0x7f54340f5410 2023-11-15T05:09:42.465 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.464+0000 7f542a7fc700 1 -- 172.21.15.154:0/3023372468 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f542400f300 con 0x7f54340f5410 2023-11-15T05:09:42.465 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:42.465 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:42.466 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 msgr2=0x7f5420032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:42.467 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f54300096a0 tx=0x7f5430006b40).stop 2023-11-15T05:09:42.467 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 msgr2=0x7f54340fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:42.467 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f5424014000 tx=0x7f542400b040).stop 2023-11-15T05:09:42.467 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 shutdown_connections 2023-11-15T05:09:42.467 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:42.468 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 --2- 172.21.15.154:0/3023372468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54340f5410 0x7f54340fd3a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:42.468 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 >> 172.21.15.154:0/3023372468 conn(0x7f54340f0b70 msgr2=0x7f543405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:42.468 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 shutdown_connections 2023-11-15T05:09:42.468 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:42.465+0000 7f543bc17700 1 -- 172.21.15.154:0/3023372468 wait complete. 2023-11-15T05:09:42.469 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:42.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:42 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:42.464978+0000 mon.smithi086 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.154:0/3023372468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:43.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:43.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:43.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:43 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:42.876306+0000 mgr.smithi086.hkfamo (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:44.912 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 -- 172.21.15.154:0/3197229464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 msgr2=0x7f84940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:44.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 --2- 172.21.15.154:0/3197229464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f84940f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f8488009fb0 tx=0x7f8488009b30).stop 2023-11-15T05:09:44.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 -- 172.21.15.154:0/3197229464 shutdown_connections 2023-11-15T05:09:44.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 --2- 172.21.15.154:0/3197229464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f84940f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:44.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 -- 172.21.15.154:0/3197229464 >> 172.21.15.154:0/3197229464 conn(0x7f84940f0b70 msgr2=0x7f84940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:44.913 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 -- 172.21.15.154:0/3197229464 shutdown_connections 2023-11-15T05:09:44.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.910+0000 7f8499844700 1 -- 172.21.15.154:0/3197229464 wait complete. 2023-11-15T05:09:44.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.911+0000 7f8499844700 1 Processor -- start 2023-11-15T05:09:44.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.911+0000 7f8499844700 1 -- start start 2023-11-15T05:09:44.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.911+0000 7f8499844700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:44.914 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.911+0000 7f8499844700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494105af0 con 0x7f84940f5410 2023-11-15T05:09:44.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8492ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:44.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8492ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42874/0 (socket says 172.21.15.154:42874) 2023-11-15T05:09:44.915 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8492ffd700 1 -- 172.21.15.154:0/2531238733 learned_addr learned my addr 172.21.15.154:0/2531238733 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:44.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8492ffd700 1 -- 172.21.15.154:0/2531238733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8488005040 con 0x7f84940f5410 2023-11-15T05:09:44.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8492ffd700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f8488013510 tx=0x7f848800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:44.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8488009320 con 0x7f84940f5410 2023-11-15T05:09:44.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8494105cf0 con 0x7f84940f5410 2023-11-15T05:09:44.917 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8488013f00 con 0x7f84940f5410 2023-11-15T05:09:44.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8488014d90 con 0x7f84940f5410 2023-11-15T05:09:44.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.912+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8494106180 con 0x7f84940f5410 2023-11-15T05:09:44.918 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.913+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8488014420 con 0x7f84940f5410 2023-11-15T05:09:44.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.913+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8494045ad0 con 0x7f84940f5410 2023-11-15T05:09:44.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.913+0000 7f847b7fe700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 0x7f847c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:44.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.913+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8488041080 con 0x7f84940f5410 2023-11-15T05:09:44.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.914+0000 7f84927fc700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 0x7f847c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:44.919 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.914+0000 7f84927fc700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 0x7f847c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f84840096a0 tx=0x7f8484006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:44.920 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:44.918+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84880095f0 con 0x7f84940f5410 2023-11-15T05:09:45.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.127+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84940fae60 con 0x7f84940f5410 2023-11-15T05:09:45.128 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.128+0000 7f847b7fe700 1 -- 172.21.15.154:0/2531238733 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f848801db70 con 0x7f84940f5410 2023-11-15T05:09:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:45.129 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:45.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 msgr2=0x7f847c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:45.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 0x7f847c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f84840096a0 tx=0x7f8484006b40).stop 2023-11-15T05:09:45.130 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 msgr2=0x7f8494105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f8488013510 tx=0x7f848800b040).stop 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 shutdown_connections 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f847c02ff60 0x7f847c032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 --2- 172.21.15.154:0/2531238733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84940f5410 0x7f8494105600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 >> 172.21.15.154:0/2531238733 conn(0x7f84940f0b70 msgr2=0x7f84940f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 shutdown_connections 2023-11-15T05:09:45.132 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:45.129+0000 7f8499844700 1 -- 172.21.15.154:0/2531238733 wait complete. 2023-11-15T05:09:45.132 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:45.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:44 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:44.876680+0000 mgr.smithi086.hkfamo (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:46.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:45 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:45.128807+0000 mon.smithi086 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.154:0/2531238733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:46.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:46.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:47.620 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:47 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:46.877118+0000 mgr.smithi086.hkfamo (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:47.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 -- 172.21.15.154:0/622139329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 msgr2=0x7f4b500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:47.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 --2- 172.21.15.154:0/622139329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b500f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4b38009fb0 tx=0x7f4b38009b30).stop 2023-11-15T05:09:47.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 -- 172.21.15.154:0/622139329 shutdown_connections 2023-11-15T05:09:47.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 --2- 172.21.15.154:0/622139329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b500f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:47.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 -- 172.21.15.154:0/622139329 >> 172.21.15.154:0/622139329 conn(0x7f4b500f0b70 msgr2=0x7f4b500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:47.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 -- 172.21.15.154:0/622139329 shutdown_connections 2023-11-15T05:09:47.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.650+0000 7f4b54abf700 1 -- 172.21.15.154:0/622139329 wait complete. 2023-11-15T05:09:47.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b54abf700 1 Processor -- start 2023-11-15T05:09:47.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b54abf700 1 -- start start 2023-11-15T05:09:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b54abf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b54abf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50106c10 con 0x7f4b500f5410 2023-11-15T05:09:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b4e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b4e59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42886/0 (socket says 172.21.15.154:42886) 2023-11-15T05:09:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b4e59c700 1 -- 172.21.15.154:0/1300547988 learned_addr learned my addr 172.21.15.154:0/1300547988 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:47.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.651+0000 7f4b4e59c700 1 -- 172.21.15.154:0/1300547988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38005040 con 0x7f4b500f5410 2023-11-15T05:09:47.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b4e59c700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4b38013510 tx=0x7f4b3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:47.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b38009320 con 0x7f4b500f5410 2023-11-15T05:09:47.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b38013f00 con 0x7f4b500f5410 2023-11-15T05:09:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b38014d90 con 0x7f4b500f5410 2023-11-15T05:09:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b50106e10 con 0x7f4b500f5410 2023-11-15T05:09:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.652+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b501072a0 con 0x7f4b500f5410 2023-11-15T05:09:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b38014420 con 0x7f4b500f5410 2023-11-15T05:09:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b46ffd700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 0x7f4b3c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:47.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b50045ad0 con 0x7f4b500f5410 2023-11-15T05:09:47.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b38041090 con 0x7f4b500f5410 2023-11-15T05:09:47.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b4dd9b700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 0x7f4b3c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:47.658 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.653+0000 7f4b4dd9b700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 0x7f4b3c032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:47.660 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.658+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b380095f0 con 0x7f4b500f5410 2023-11-15T05:09:47.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.864+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b50108150 con 0x7f4b500f5410 2023-11-15T05:09:47.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.865+0000 7f4b46ffd700 1 -- 172.21.15.154:0/1300547988 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b3802b020 con 0x7f4b500f5410 2023-11-15T05:09:47.866 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:47.866 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:47.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 msgr2=0x7f4b3c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:47.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 0x7f4b3c032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).stop 2023-11-15T05:09:47.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 msgr2=0x7f4b50106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:47.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4b38013510 tx=0x7f4b3800b040).stop 2023-11-15T05:09:47.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 shutdown_connections 2023-11-15T05:09:47.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4b3c02ffb0 0x7f4b3c032470 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:47.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 --2- 172.21.15.154:0/1300547988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b500f5410 0x7f4b50106720 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:47.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 >> 172.21.15.154:0/1300547988 conn(0x7f4b500f0b70 msgr2=0x7f4b5005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:47.868 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 shutdown_connections 2023-11-15T05:09:47.869 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:47.866+0000 7f4b54abf700 1 -- 172.21.15.154:0/1300547988 wait complete. 2023-11-15T05:09:47.869 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:48.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:48 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:47.865877+0000 mon.smithi086 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.154:0/1300547988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:49.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:49.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:49.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:49 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:48.877469+0000 mgr.smithi086.hkfamo (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:50.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.419+0000 7f64f338f700 1 -- 172.21.15.154:0/3049980005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 msgr2=0x7f64ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:50.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.419+0000 7f64f338f700 1 --2- 172.21.15.154:0/3049980005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec0f55c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f64dc009fb0 tx=0x7f64dc009b30).stop 2023-11-15T05:09:50.422 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- 172.21.15.154:0/3049980005 shutdown_connections 2023-11-15T05:09:50.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 --2- 172.21.15.154:0/3049980005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec0f55c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:50.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- 172.21.15.154:0/3049980005 >> 172.21.15.154:0/3049980005 conn(0x7f64ec0f09a0 msgr2=0x7f64ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:50.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- 172.21.15.154:0/3049980005 shutdown_connections 2023-11-15T05:09:50.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- 172.21.15.154:0/3049980005 wait complete. 2023-11-15T05:09:50.423 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 Processor -- start 2023-11-15T05:09:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- start start 2023-11-15T05:09:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f338f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec1058a0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f238d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f238d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42902/0 (socket says 172.21.15.154:42902) 2023-11-15T05:09:50.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.420+0000 7f64f238d700 1 -- 172.21.15.154:0/2741797394 learned_addr learned my addr 172.21.15.154:0/2741797394 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:50.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64f238d700 1 -- 172.21.15.154:0/2741797394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc005040 con 0x7f64ec0f51a0 2023-11-15T05:09:50.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64f238d700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f64dc013510 tx=0x7f64dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:50.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64dc009320 con 0x7f64ec0f51a0 2023-11-15T05:09:50.425 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec105aa0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64dc013f00 con 0x7f64ec0f51a0 2023-11-15T05:09:50.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64dc014dc0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.426 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.421+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec105f30 con 0x7f64ec0f51a0 2023-11-15T05:09:50.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.422+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec045ad0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.427 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.422+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64dc0095f0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.422+0000 7f64e2ffd700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:50.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.422+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64dc0410c0 con 0x7f64ec0f51a0 2023-11-15T05:09:50.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.422+0000 7f64f1b8c700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:50.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.423+0000 7f64f1b8c700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 0x7f64d8032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f64e80096a0 tx=0x7f64e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:50.428 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.427+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64dc01c790 con 0x7f64ec0f51a0 2023-11-15T05:09:50.635 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.634+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64ec106d80 con 0x7f64ec0f51a0 2023-11-15T05:09:50.636 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.635+0000 7f64e2ffd700 1 -- 172.21.15.154:0/2741797394 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64dc01c790 con 0x7f64ec0f51a0 2023-11-15T05:09:50.636 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:50.636 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:50.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 msgr2=0x7f64d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:50.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 0x7f64d8032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f64e80096a0 tx=0x7f64e8006b40).stop 2023-11-15T05:09:50.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 msgr2=0x7f64ec1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:50.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f64dc013510 tx=0x7f64dc00b040).stop 2023-11-15T05:09:50.638 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 shutdown_connections 2023-11-15T05:09:50.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f64d802ff60 0x7f64d8032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:50.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 --2- 172.21.15.154:0/2741797394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64ec0f51a0 0x7f64ec1053b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:50.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 >> 172.21.15.154:0/2741797394 conn(0x7f64ec0f09a0 msgr2=0x7f64ec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:50.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 shutdown_connections 2023-11-15T05:09:50.639 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:50.637+0000 7f64f338f700 1 -- 172.21.15.154:0/2741797394 wait complete. 2023-11-15T05:09:50.640 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:50.993 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:50 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:50.636170+0000 mon.smithi086 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.154:0/2741797394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:51.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:51 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:50.877825+0000 mgr.smithi086.hkfamo (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:52.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:52.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:53.201 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 -- 172.21.15.154:0/3707338146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 msgr2=0x7f4f5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:53.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 --2- 172.21.15.154:0/3707338146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f4f50009fb0 tx=0x7f4f50009b30).stop 2023-11-15T05:09:53.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 -- 172.21.15.154:0/3707338146 shutdown_connections 2023-11-15T05:09:53.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 --2- 172.21.15.154:0/3707338146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:53.202 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 -- 172.21.15.154:0/3707338146 >> 172.21.15.154:0/3707338146 conn(0x7f4f5c0f0b70 msgr2=0x7f4f5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:53.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 -- 172.21.15.154:0/3707338146 shutdown_connections 2023-11-15T05:09:53.203 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 -- 172.21.15.154:0/3707338146 wait complete. 2023-11-15T05:09:53.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.200+0000 7f4f635e5700 1 Processor -- start 2023-11-15T05:09:53.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f635e5700 1 -- start start 2023-11-15T05:09:53.205 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f635e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:53.206 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f635e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c060220 con 0x7f4f5c0f5410 2023-11-15T05:09:53.206 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f61381700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:53.206 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f61381700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:42924/0 (socket says 172.21.15.154:42924) 2023-11-15T05:09:53.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f61381700 1 -- 172.21.15.154:0/2683118819 learned_addr learned my addr 172.21.15.154:0/2683118819 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:53.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f61381700 1 -- 172.21.15.154:0/2683118819 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50005040 con 0x7f4f5c0f5410 2023-11-15T05:09:53.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.201+0000 7f4f61381700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4f50013510 tx=0x7f4f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:53.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.202+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f50009320 con 0x7f4f5c0f5410 2023-11-15T05:09:53.207 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.202+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f50013f00 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.202+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c060420 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.202+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f50014d90 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.202+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c109b00 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c059100 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f500095f0 con 0x7f4f5c0f5410 2023-11-15T05:09:53.208 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f4dffb700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 0x7f4f48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:53.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f50041000 con 0x7f4f5c0f5410 2023-11-15T05:09:53.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f60b80700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 0x7f4f48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:53.209 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.203+0000 7f4f60b80700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 0x7f4f48032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4f580096a0 tx=0x7f4f58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:53.210 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.207+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f5001d7f0 con 0x7f4f5c0f5410 2023-11-15T05:09:53.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:52 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:52.878174+0000 mgr.smithi086.hkfamo (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:53.415 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.412+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f5c045ad0 con 0x7f4f5c0f5410 2023-11-15T05:09:53.415 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f4dffb700 1 -- 172.21.15.154:0/2683118819 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f50010230 con 0x7f4f5c0f5410 2023-11-15T05:09:53.415 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:53.416 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:53.416 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 msgr2=0x7f4f48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:53.417 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 0x7f4f48032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4f580096a0 tx=0x7f4f58006b40).stop 2023-11-15T05:09:53.417 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 msgr2=0x7f4f5c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:53.417 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4f50013510 tx=0x7f4f5000b040).stop 2023-11-15T05:09:53.417 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 shutdown_connections 2023-11-15T05:09:53.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4f4802ff60 0x7f4f48032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:53.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 --2- 172.21.15.154:0/2683118819 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f5c0f5410 0x7f4f5c109450 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:53.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 >> 172.21.15.154:0/2683118819 conn(0x7f4f5c0f0b70 msgr2=0x7f4f5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:53.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 shutdown_connections 2023-11-15T05:09:53.418 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:53.414+0000 7f4f635e5700 1 -- 172.21.15.154:0/2683118819 wait complete. 2023-11-15T05:09:53.419 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:54.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:53 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:53.414840+0000 mon.smithi086 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.154:0/2683118819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:54.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:54.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:55.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:54 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:54.878552+0000 mgr.smithi086.hkfamo (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 -- 172.21.15.154:0/3770836078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 msgr2=0x7f389c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 --2- 172.21.15.154:0/3770836078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f3890009fb0 tx=0x7f3890009b30).stop 2023-11-15T05:09:55.974 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 -- 172.21.15.154:0/3770836078 shutdown_connections 2023-11-15T05:09:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 --2- 172.21.15.154:0/3770836078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 -- 172.21.15.154:0/3770836078 >> 172.21.15.154:0/3770836078 conn(0x7f389c0f0b70 msgr2=0x7f389c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 -- 172.21.15.154:0/3770836078 shutdown_connections 2023-11-15T05:09:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.970+0000 7f38a2e27700 1 -- 172.21.15.154:0/3770836078 wait complete. 2023-11-15T05:09:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a2e27700 1 Processor -- start 2023-11-15T05:09:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a2e27700 1 -- start start 2023-11-15T05:09:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a2e27700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a2e27700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c106bd0 con 0x7f389c0f5410 2023-11-15T05:09:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a0bc3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:55.976 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a0bc3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53168/0 (socket says 172.21.15.154:53168) 2023-11-15T05:09:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.971+0000 7f38a0bc3700 1 -- 172.21.15.154:0/2947653652 learned_addr learned my addr 172.21.15.154:0/2947653652 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38a0bc3700 1 -- 172.21.15.154:0/2947653652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3890005040 con 0x7f389c0f5410 2023-11-15T05:09:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38a0bc3700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3890013510 tx=0x7f389000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3890009320 con 0x7f389c0f5410 2023-11-15T05:09:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c106dd0 con 0x7f389c0f5410 2023-11-15T05:09:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3890013f00 con 0x7f389c0f5410 2023-11-15T05:09:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3890014d90 con 0x7f389c0f5410 2023-11-15T05:09:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.972+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c107260 con 0x7f389c0f5410 2023-11-15T05:09:55.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.973+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3890014420 con 0x7f389c0f5410 2023-11-15T05:09:55.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.973+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c045ad0 con 0x7f389c0f5410 2023-11-15T05:09:55.981 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.973+0000 7f38997fa700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 0x7f388c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.973+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38900410c0 con 0x7f389c0f5410 2023-11-15T05:09:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.974+0000 7f389bfff700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 0x7f388c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.974+0000 7f389bfff700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 0x7f388c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f38880096a0 tx=0x7f3888006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:55.979+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38900095f0 con 0x7f389c0f5410 2023-11-15T05:09:56.185 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.184+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f389c0fae60 con 0x7f389c0f5410 2023-11-15T05:09:56.186 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.184+0000 7f38997fa700 1 -- 172.21.15.154:0/2947653652 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f389001db70 con 0x7f389c0f5410 2023-11-15T05:09:56.187 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:56.187 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:56.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 msgr2=0x7f388c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:56.188 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 0x7f388c032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f38880096a0 tx=0x7f3888006b40).stop 2023-11-15T05:09:56.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 msgr2=0x7f389c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:56.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3890013510 tx=0x7f389000b040).stop 2023-11-15T05:09:56.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 shutdown_connections 2023-11-15T05:09:56.189 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f388c02ff60 0x7f388c032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:56.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 --2- 172.21.15.154:0/2947653652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f389c0f5410 0x7f389c1066e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:56.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 >> 172.21.15.154:0/2947653652 conn(0x7f389c0f0b70 msgr2=0x7f389c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:56.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 shutdown_connections 2023-11-15T05:09:56.190 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:56.186+0000 7f38a2e27700 1 -- 172.21.15.154:0/2947653652 wait complete. 2023-11-15T05:09:56.191 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:56.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:56 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:56.186440+0000 mon.smithi086 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.154:0/2947653652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:09:57.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:09:57.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:09:57.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:57 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:56.878941+0000 mgr.smithi086.hkfamo (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:58.720 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 -- 172.21.15.154:0/2171719423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 msgr2=0x7f75580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 --2- 172.21.15.154:0/2171719423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f75580f6930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f754c009fb0 tx=0x7f754c009b30).stop 2023-11-15T05:09:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 -- 172.21.15.154:0/2171719423 shutdown_connections 2023-11-15T05:09:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 --2- 172.21.15.154:0/2171719423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f75580f6930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 -- 172.21.15.154:0/2171719423 >> 172.21.15.154:0/2171719423 conn(0x7f75580f1c70 msgr2=0x7f75580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:58.721 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 -- 172.21.15.154:0/2171719423 shutdown_connections 2023-11-15T05:09:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.719+0000 7f755eff9700 1 -- 172.21.15.154:0/2171719423 wait complete. 2023-11-15T05:09:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755eff9700 1 Processor -- start 2023-11-15T05:09:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755eff9700 1 -- start start 2023-11-15T05:09:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755eff9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755eff9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7558106c20 con 0x7f75580f6510 2023-11-15T05:09:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755cd95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755cd95700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53182/0 (socket says 172.21.15.154:53182) 2023-11-15T05:09:58.724 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.720+0000 7f755cd95700 1 -- 172.21.15.154:0/46102420 learned_addr learned my addr 172.21.15.154:0/46102420 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:09:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f755cd95700 1 -- 172.21.15.154:0/46102420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c005040 con 0x7f75580f6510 2023-11-15T05:09:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f755cd95700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f754c013510 tx=0x7f754c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f754c009320 con 0x7f75580f6510 2023-11-15T05:09:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7558106e20 con 0x7f75580f6510 2023-11-15T05:09:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f754c013f00 con 0x7f75580f6510 2023-11-15T05:09:58.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f754c014d90 con 0x7f75580f6510 2023-11-15T05:09:58.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.721+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75581072b0 con 0x7f75580f6510 2023-11-15T05:09:58.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.722+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f754c0263a0 con 0x7f75580f6510 2023-11-15T05:09:58.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.722+0000 7f75557fa700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 0x7f7548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:09:58.726 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.722+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f754c041030 con 0x7f75580f6510 2023-11-15T05:09:58.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.722+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7558059100 con 0x7f75580f6510 2023-11-15T05:09:58.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.722+0000 7f7557fff700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 0x7f7548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:09:58.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.723+0000 7f7557fff700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 0x7f7548032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f75440096a0 tx=0x7f7544006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:09:58.727 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.727+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f754c012920 con 0x7f75580f6510 2023-11-15T05:09:58.938 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.937+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75580fbf60 con 0x7f75580f6510 2023-11-15T05:09:58.939 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.938+0000 7f75557fa700 1 -- 172.21.15.154:0/46102420 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f754c02b020 con 0x7f75580f6510 2023-11-15T05:09:58.939 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:09:58.939 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:09:58.940 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.939+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 msgr2=0x7f7548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:58.940 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.939+0000 7f755eff9700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 0x7f7548032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f75440096a0 tx=0x7f7544006b40).stop 2023-11-15T05:09:58.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.939+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 msgr2=0x7f7558106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:09:58.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.939+0000 7f755eff9700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f754c013510 tx=0x7f754c00b040).stop 2023-11-15T05:09:58.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 shutdown_connections 2023-11-15T05:09:58.941 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f754802ff60 0x7f7548032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:58.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 --2- 172.21.15.154:0/46102420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75580f6510 0x7f7558106730 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:09:58.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 >> 172.21.15.154:0/46102420 conn(0x7f75580f1c70 msgr2=0x7f75580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:09:58.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 shutdown_connections 2023-11-15T05:09:58.942 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:09:58.940+0000 7f755eff9700 1 -- 172.21.15.154:0/46102420 wait complete. 2023-11-15T05:09:58.943 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:09:59.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:09:58.879270+0000 mgr.smithi086.hkfamo (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:09:59.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:09:58 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:09:58.939065+0000 mon.smithi086 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.154:0/46102420' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:00.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:00.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:01.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:00 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:00.879620+0000 mgr.smithi086.hkfamo (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:01.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1538099706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 msgr2=0x7fdc240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:01.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1538099706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc240f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fdc18009fb0 tx=0x7fdc18009b30).stop 2023-11-15T05:10:01.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1538099706 shutdown_connections 2023-11-15T05:10:01.490 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1538099706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc240f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:01.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1538099706 >> 172.21.15.154:0/1538099706 conn(0x7fdc240f0b70 msgr2=0x7fdc240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:01.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1538099706 shutdown_connections 2023-11-15T05:10:01.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.488+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1538099706 wait complete. 2023-11-15T05:10:01.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc2b0e5700 1 Processor -- start 2023-11-15T05:10:01.491 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc2b0e5700 1 -- start start 2023-11-15T05:10:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc2b0e5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc2b0e5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc24107240 con 0x7fdc240f5410 2023-11-15T05:10:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc28e81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc28e81700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:53208/0 (socket says 172.21.15.154:53208) 2023-11-15T05:10:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.489+0000 7fdc28e81700 1 -- 172.21.15.154:0/1403165635 learned_addr learned my addr 172.21.15.154:0/1403165635 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:01.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc28e81700 1 -- 172.21.15.154:0/1403165635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18005040 con 0x7fdc240f5410 2023-11-15T05:10:01.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc28e81700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fdc18013510 tx=0x7fdc1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:01.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc18009320 con 0x7fdc240f5410 2023-11-15T05:10:01.494 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc24060220 con 0x7fdc240f5410 2023-11-15T05:10:01.495 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc18013f00 con 0x7fdc240f5410 2023-11-15T05:10:01.495 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc18014d90 con 0x7fdc240f5410 2023-11-15T05:10:01.495 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.490+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2410bd40 con 0x7fdc240f5410 2023-11-15T05:10:01.495 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.491+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc180095f0 con 0x7fdc240f5410 2023-11-15T05:10:01.495 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.491+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc24045ad0 con 0x7fdc240f5410 2023-11-15T05:10:01.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.491+0000 7fdc217fa700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 0x7fdc14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:01.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.491+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc18041090 con 0x7fdc240f5410 2023-11-15T05:10:01.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.491+0000 7fdc23fff700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 0x7fdc14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:01.496 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.492+0000 7fdc23fff700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 0x7fdc14032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fdc100096a0 tx=0x7fdc10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:01.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.496+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc18014420 con 0x7fdc240f5410 2023-11-15T05:10:01.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.701+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc240fae60 con 0x7fdc240f5410 2023-11-15T05:10:01.702 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.702+0000 7fdc217fa700 1 -- 172.21.15.154:0/1403165635 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc1801d7f0 con 0x7fdc240f5410 2023-11-15T05:10:01.703 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:01.703 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:01.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 msgr2=0x7fdc14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:01.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 0x7fdc14032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fdc100096a0 tx=0x7fdc10006b40).stop 2023-11-15T05:10:01.704 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 msgr2=0x7fdc2410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fdc18013510 tx=0x7fdc1800b040).stop 2023-11-15T05:10:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 shutdown_connections 2023-11-15T05:10:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fdc1402ff60 0x7fdc14032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 --2- 172.21.15.154:0/1403165635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc240f5410 0x7fdc2410b850 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 >> 172.21.15.154:0/1403165635 conn(0x7fdc240f0b70 msgr2=0x7fdc240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:01.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 shutdown_connections 2023-11-15T05:10:01.706 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:01.704+0000 7fdc2b0e5700 1 -- 172.21.15.154:0/1403165635 wait complete. 2023-11-15T05:10:01.706 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:02.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:02 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:01.703082+0000 mon.smithi086 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.154:0/1403165635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:03.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:03.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:03.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:03 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:02.879875+0000 mgr.smithi086.hkfamo (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:04.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 -- 172.21.15.154:0/1596632051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 msgr2=0x7f13480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:04.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 --2- 172.21.15.154:0/1596632051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f13480f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f1330009fb0 tx=0x7f1330009b30).stop 2023-11-15T05:10:04.285 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 -- 172.21.15.154:0/1596632051 shutdown_connections 2023-11-15T05:10:04.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 --2- 172.21.15.154:0/1596632051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f13480f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:04.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 -- 172.21.15.154:0/1596632051 >> 172.21.15.154:0/1596632051 conn(0x7f13480f1c70 msgr2=0x7f13480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:04.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 -- 172.21.15.154:0/1596632051 shutdown_connections 2023-11-15T05:10:04.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.282+0000 7f134d3a5700 1 -- 172.21.15.154:0/1596632051 wait complete. 2023-11-15T05:10:04.286 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f134d3a5700 1 Processor -- start 2023-11-15T05:10:04.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f134d3a5700 1 -- start start 2023-11-15T05:10:04.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f134d3a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:04.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f134d3a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348069e20 con 0x7f13480f6510 2023-11-15T05:10:04.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f1346ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:04.287 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f1346ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43810/0 (socket says 172.21.15.154:43810) 2023-11-15T05:10:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.283+0000 7f1346ffd700 1 -- 172.21.15.154:0/3842148456 learned_addr learned my addr 172.21.15.154:0/3842148456 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f1346ffd700 1 -- 172.21.15.154:0/3842148456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330005040 con 0x7f13480f6510 2023-11-15T05:10:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f1346ffd700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f1330013510 tx=0x7f133000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1330009320 con 0x7f13480f6510 2023-11-15T05:10:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348069fc0 con 0x7f13480f6510 2023-11-15T05:10:04.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1330013f00 con 0x7f13480f6510 2023-11-15T05:10:04.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1330014d90 con 0x7f13480f6510 2023-11-15T05:10:04.292 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.284+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134806a450 con 0x7f13480f6510 2023-11-15T05:10:04.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.285+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13300263a0 con 0x7f13480f6510 2023-11-15T05:10:04.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.285+0000 7f133f7fe700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:04.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.285+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1330041010 con 0x7f13480f6510 2023-11-15T05:10:04.294 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.285+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348059100 con 0x7f13480f6510 2023-11-15T05:10:04.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.285+0000 7f13467fc700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:04.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.286+0000 7f13467fc700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 0x7f1334032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f13380096a0 tx=0x7f1338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:04.295 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.290+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1330012920 con 0x7f13480f6510 2023-11-15T05:10:04.497 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.497+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f134806ab50 con 0x7f13480f6510 2023-11-15T05:10:04.499 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.497+0000 7f133f7fe700 1 -- 172.21.15.154:0/3842148456 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f133002b020 con 0x7f13480f6510 2023-11-15T05:10:04.499 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:04.500 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:04.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 msgr2=0x7f1334032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:04.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 0x7f1334032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f13380096a0 tx=0x7f1338006b40).stop 2023-11-15T05:10:04.501 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 msgr2=0x7f134806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:04.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f1330013510 tx=0x7f133000b040).stop 2023-11-15T05:10:04.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 shutdown_connections 2023-11-15T05:10:04.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f133402ff60 0x7f1334032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:04.502 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 --2- 172.21.15.154:0/3842148456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13480f6510 0x7f134806b9a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:04.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 >> 172.21.15.154:0/3842148456 conn(0x7f13480f1c70 msgr2=0x7f13480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:04.503 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 shutdown_connections 2023-11-15T05:10:04.504 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:04.499+0000 7f134d3a5700 1 -- 172.21.15.154:0/3842148456 wait complete. 2023-11-15T05:10:04.504 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:04.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:04 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:04.498469+0000 mon.smithi086 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.154:0/3842148456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:05.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:05.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:05.989 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:05 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:04.880211+0000 mgr.smithi086.hkfamo (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:07.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 -- 172.21.15.154:0/3606140586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 msgr2=0x7f22300fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:07.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 --2- 172.21.15.154:0/3606140586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22300fb570 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f2224009fb0 tx=0x7f2224009b30).stop 2023-11-15T05:10:07.058 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 -- 172.21.15.154:0/3606140586 shutdown_connections 2023-11-15T05:10:07.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 --2- 172.21.15.154:0/3606140586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22300fb570 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:07.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 -- 172.21.15.154:0/3606140586 >> 172.21.15.154:0/3606140586 conn(0x7f22300f0b70 msgr2=0x7f22300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:07.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.056+0000 7f2236b05700 1 -- 172.21.15.154:0/3606140586 shutdown_connections 2023-11-15T05:10:07.059 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.057+0000 7f2236b05700 1 -- 172.21.15.154:0/3606140586 wait complete. 2023-11-15T05:10:07.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.057+0000 7f2236b05700 1 Processor -- start 2023-11-15T05:10:07.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.057+0000 7f2236b05700 1 -- start start 2023-11-15T05:10:07.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.057+0000 7f2236b05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:07.060 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.057+0000 7f2236b05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230108dc0 con 0x7f2230060200 2023-11-15T05:10:07.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f22348a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:07.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f22348a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43828/0 (socket says 172.21.15.154:43828) 2023-11-15T05:10:07.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f22348a1700 1 -- 172.21.15.154:0/162948215 learned_addr learned my addr 172.21.15.154:0/162948215 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:07.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f22348a1700 1 -- 172.21.15.154:0/162948215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224005040 con 0x7f2230060200 2023-11-15T05:10:07.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f22348a1700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f2224013510 tx=0x7f222400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:07.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2224009320 con 0x7f2230060200 2023-11-15T05:10:07.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2230108fc0 con 0x7f2230060200 2023-11-15T05:10:07.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2224013f00 con 0x7f2230060200 2023-11-15T05:10:07.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.058+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2224014dc0 con 0x7f2230060200 2023-11-15T05:10:07.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.059+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2230109450 con 0x7f2230060200 2023-11-15T05:10:07.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.059+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f222401d3d0 con 0x7f2230060200 2023-11-15T05:10:07.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.059+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2230045ad0 con 0x7f2230060200 2023-11-15T05:10:07.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.059+0000 7f222d7fa700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 0x7f2220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:07.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.059+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22240410e0 con 0x7f2230060200 2023-11-15T05:10:07.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.060+0000 7f222ffff700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 0x7f2220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:07.068 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.060+0000 7f222ffff700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 0x7f2220032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f221c0096a0 tx=0x7f221c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:07.069 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.064+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22240126f0 con 0x7f2230060200 2023-11-15T05:10:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.271+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f223010a300 con 0x7f2230060200 2023-11-15T05:10:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.272+0000 7f222d7fa700 1 -- 172.21.15.154:0/162948215 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2224012330 con 0x7f2230060200 2023-11-15T05:10:07.275 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:07.275 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 msgr2=0x7f2220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 0x7f2220032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f221c0096a0 tx=0x7f221c006b40).stop 2023-11-15T05:10:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 msgr2=0x7f22301088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f2224013510 tx=0x7f222400b040).stop 2023-11-15T05:10:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 shutdown_connections 2023-11-15T05:10:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f222002ff60 0x7f2220032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 --2- 172.21.15.154:0/162948215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2230060200 0x7f22301088d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 >> 172.21.15.154:0/162948215 conn(0x7f22300f0b70 msgr2=0x7f22300f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.273+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 shutdown_connections 2023-11-15T05:10:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:07.274+0000 7f2236b05700 1 -- 172.21.15.154:0/162948215 wait complete. 2023-11-15T05:10:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:07.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:06.880544+0000 mgr.smithi086.hkfamo (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:07.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:07 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:07.273023+0000 mon.smithi086 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.154:0/162948215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:08.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:08.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:09.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:08 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:08.880885+0000 mgr.smithi086.hkfamo (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:09.851 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/2495842084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 msgr2=0x7fc7e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:09.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/2495842084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e80f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fc7d0009fb0 tx=0x7fc7d0009b30).stop 2023-11-15T05:10:09.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/2495842084 shutdown_connections 2023-11-15T05:10:09.852 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/2495842084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e80f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:09.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/2495842084 >> 172.21.15.154:0/2495842084 conn(0x7fc7e80f0b70 msgr2=0x7fc7e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:09.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/2495842084 shutdown_connections 2023-11-15T05:10:09.853 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/2495842084 wait complete. 2023-11-15T05:10:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 Processor -- start 2023-11-15T05:10:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.851+0000 7fc7ecbc8700 1 -- start start 2023-11-15T05:10:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7ecbc8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7ecbc8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e8060240 con 0x7fc7e80f5410 2023-11-15T05:10:09.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7e659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:09.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7e659c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43842/0 (socket says 172.21.15.154:43842) 2023-11-15T05:10:09.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7e659c700 1 -- 172.21.15.154:0/1491778196 learned_addr learned my addr 172.21.15.154:0/1491778196 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:09.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7e659c700 1 -- 172.21.15.154:0/1491778196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d0005040 con 0x7fc7e80f5410 2023-11-15T05:10:09.858 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7e659c700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fc7d0013510 tx=0x7fc7d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:09.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.852+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d0009320 con 0x7fc7e80f5410 2023-11-15T05:10:09.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e8060440 con 0x7fc7e80f5410 2023-11-15T05:10:09.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7d0013f00 con 0x7fc7e80f5410 2023-11-15T05:10:09.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d0014d90 con 0x7fc7e80f5410 2023-11-15T05:10:09.859 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e8109ab0 con 0x7fc7e80f5410 2023-11-15T05:10:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7d0014420 con 0x7fc7e80f5410 2023-11-15T05:10:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e8045ad0 con 0x7fc7e80f5410 2023-11-15T05:10:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.853+0000 7fc7deffd700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 0x7fc7d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.854+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7d00410c0 con 0x7fc7e80f5410 2023-11-15T05:10:09.860 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.854+0000 7fc7e5d9b700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 0x7fc7d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.854+0000 7fc7e5d9b700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 0x7fc7d4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc7d80096a0 tx=0x7fc7d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:09.861 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:09.858+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7d00095f0 con 0x7fc7e80f5410 2023-11-15T05:10:10.063 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.063+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7e80fae60 con 0x7fc7e80f5410 2023-11-15T05:10:10.064 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.063+0000 7fc7deffd700 1 -- 172.21.15.154:0/1491778196 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7d001db70 con 0x7fc7e80f5410 2023-11-15T05:10:10.064 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:10.064 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:10.065 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 msgr2=0x7fc7d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:10.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 0x7fc7d4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc7d80096a0 tx=0x7fc7d8006b40).stop 2023-11-15T05:10:10.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 msgr2=0x7fc7e8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:10.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fc7d0013510 tx=0x7fc7d000b040).stop 2023-11-15T05:10:10.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 shutdown_connections 2023-11-15T05:10:10.066 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fc7d402ff60 0x7fc7d4032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 --2- 172.21.15.154:0/1491778196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7e80f5410 0x7fc7e8109400 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 >> 172.21.15.154:0/1491778196 conn(0x7fc7e80f0b70 msgr2=0x7fc7e80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 shutdown_connections 2023-11-15T05:10:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:10.065+0000 7fc7ecbc8700 1 -- 172.21.15.154:0/1491778196 wait complete. 2023-11-15T05:10:10.068 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:10.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:10 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:10.064376+0000 mon.smithi086 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.154:0/1491778196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:11.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:11.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:11.489 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:11 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:10.881245+0000 mgr.smithi086.hkfamo (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:12.667 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.665+0000 7f4fec21c700 1 -- 172.21.15.154:0/1698498645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 msgr2=0x7f4fe40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:12.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.665+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1698498645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe40f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f4fd4009fb0 tx=0x7f4fd4009b30).stop 2023-11-15T05:10:12.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.665+0000 7f4fec21c700 1 -- 172.21.15.154:0/1698498645 shutdown_connections 2023-11-15T05:10:12.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.665+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1698498645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe40f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:12.668 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.665+0000 7f4fec21c700 1 -- 172.21.15.154:0/1698498645 >> 172.21.15.154:0/1698498645 conn(0x7f4fe40f1c70 msgr2=0x7f4fe40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:12.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 -- 172.21.15.154:0/1698498645 shutdown_connections 2023-11-15T05:10:12.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 -- 172.21.15.154:0/1698498645 wait complete. 2023-11-15T05:10:12.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 Processor -- start 2023-11-15T05:10:12.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 -- start start 2023-11-15T05:10:12.669 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:12.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.666+0000 7f4fec21c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe4069e20 con 0x7f4fe40f6510 2023-11-15T05:10:12.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fe9fb8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:12.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fe9fb8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:43850/0 (socket says 172.21.15.154:43850) 2023-11-15T05:10:12.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fe9fb8700 1 -- 172.21.15.154:0/1270008563 learned_addr learned my addr 172.21.15.154:0/1270008563 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:12.670 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fe9fb8700 1 -- 172.21.15.154:0/1270008563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4005040 con 0x7f4fe40f6510 2023-11-15T05:10:12.671 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fe9fb8700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4fd4013510 tx=0x7f4fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:12.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fd4009320 con 0x7f4fe40f6510 2023-11-15T05:10:12.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe4069fc0 con 0x7f4fe40f6510 2023-11-15T05:10:12.672 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fd4013f00 con 0x7f4fe40f6510 2023-11-15T05:10:12.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fd4014d90 con 0x7f4fe40f6510 2023-11-15T05:10:12.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.667+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe406a450 con 0x7f4fe40f6510 2023-11-15T05:10:12.673 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.668+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fd40263a0 con 0x7f4fe40f6510 2023-11-15T05:10:12.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.668+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe4059100 con 0x7f4fe40f6510 2023-11-15T05:10:12.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.669+0000 7f4fda7fc700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 0x7f4fd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:12.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.669+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fd40412f0 con 0x7f4fe40f6510 2023-11-15T05:10:12.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.669+0000 7f4fe97b7700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 0x7f4fd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:12.674 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.669+0000 7f4fe97b7700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 0x7f4fd0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4fe00096a0 tx=0x7f4fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:12.675 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.673+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fd40126f0 con 0x7f4fe40f6510 2023-11-15T05:10:12.878 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.877+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fe40fbf60 con 0x7f4fe40f6510 2023-11-15T05:10:12.879 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.878+0000 7f4fda7fc700 1 -- 172.21.15.154:0/1270008563 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4fd4012330 con 0x7f4fe40f6510 2023-11-15T05:10:12.879 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:12.879 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:12.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 msgr2=0x7f4fd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:12.880 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 0x7f4fd0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4fe00096a0 tx=0x7f4fe0006b40).stop 2023-11-15T05:10:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 msgr2=0x7f4fe406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f4fd4013510 tx=0x7f4fd400b040).stop 2023-11-15T05:10:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 shutdown_connections 2023-11-15T05:10:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f4fd002ff60 0x7f4fd0032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 --2- 172.21.15.154:0/1270008563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fe40f6510 0x7f4fe406b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:12.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 >> 172.21.15.154:0/1270008563 conn(0x7f4fe40f1c70 msgr2=0x7f4fe40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:12.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 shutdown_connections 2023-11-15T05:10:12.885 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:12.880+0000 7f4fec21c700 1 -- 172.21.15.154:0/1270008563 wait complete. 2023-11-15T05:10:12.886 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:13.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:12.879195+0000 mon.smithi086 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.154:0/1270008563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:13.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:12 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:12.881535+0000 mgr.smithi086.hkfamo (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:14.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:14.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:15.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:14 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:14.881841+0000 mgr.smithi086.hkfamo (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:15.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- 172.21.15.154:0/2068326078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 msgr2=0x7f06980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:15.435 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 --2- 172.21.15.154:0/2068326078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06980f66c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f068c009fb0 tx=0x7f068c009b30).stop 2023-11-15T05:10:15.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- 172.21.15.154:0/2068326078 shutdown_connections 2023-11-15T05:10:15.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 --2- 172.21.15.154:0/2068326078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06980f66c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:15.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- 172.21.15.154:0/2068326078 >> 172.21.15.154:0/2068326078 conn(0x7f06980f1aa0 msgr2=0x7f06980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:15.436 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- 172.21.15.154:0/2068326078 shutdown_connections 2023-11-15T05:10:15.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- 172.21.15.154:0/2068326078 wait complete. 2023-11-15T05:10:15.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 Processor -- start 2023-11-15T05:10:15.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.434+0000 7f069da8b700 1 -- start start 2023-11-15T05:10:15.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069da8b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:15.437 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069da8b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06981069a0 con 0x7f06980f62a0 2023-11-15T05:10:15.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069ca89700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:15.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069ca89700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:35194/0 (socket says 172.21.15.154:35194) 2023-11-15T05:10:15.438 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069ca89700 1 -- 172.21.15.154:0/2277124621 learned_addr learned my addr 172.21.15.154:0/2277124621 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:15.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069ca89700 1 -- 172.21.15.154:0/2277124621 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c005040 con 0x7f06980f62a0 2023-11-15T05:10:15.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f069ca89700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f068c013510 tx=0x7f068c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:15.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.435+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f068c009320 con 0x7f06980f62a0 2023-11-15T05:10:15.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f068c013f00 con 0x7f06980f62a0 2023-11-15T05:10:15.440 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0698106ba0 con 0x7f06980f62a0 2023-11-15T05:10:15.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f068c014dc0 con 0x7f06980f62a0 2023-11-15T05:10:15.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0698107030 con 0x7f06980f62a0 2023-11-15T05:10:15.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f068c014420 con 0x7f06980f62a0 2023-11-15T05:10:15.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0698045ad0 con 0x7f06980f62a0 2023-11-15T05:10:15.441 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.436+0000 7f06957fa700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 0x7f0688032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:15.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.437+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f068c041110 con 0x7f06980f62a0 2023-11-15T05:10:15.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.437+0000 7f0697fff700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 0x7f0688032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:15.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.437+0000 7f0697fff700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 0x7f0688032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f06840096a0 tx=0x7f0684006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:15.442 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.441+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f068c0095f0 con 0x7f06980f62a0 2023-11-15T05:10:15.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.648+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0698107ee0 con 0x7f06980f62a0 2023-11-15T05:10:15.649 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.649+0000 7f06957fa700 1 -- 172.21.15.154:0/2277124621 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f068c01cb10 con 0x7f06980f62a0 2023-11-15T05:10:15.650 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:15.650 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:15.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 msgr2=0x7f0688032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:15.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 0x7f0688032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f06840096a0 tx=0x7f0684006b40).stop 2023-11-15T05:10:15.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 msgr2=0x7f06981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:15.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f068c013510 tx=0x7f068c00b040).stop 2023-11-15T05:10:15.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 shutdown_connections 2023-11-15T05:10:15.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f068802ff60 0x7f0688032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:15.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 --2- 172.21.15.154:0/2277124621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06980f62a0 0x7f06981064b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:15.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 >> 172.21.15.154:0/2277124621 conn(0x7f06980f1aa0 msgr2=0x7f06980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:15.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 shutdown_connections 2023-11-15T05:10:15.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:15.650+0000 7f069da8b700 1 -- 172.21.15.154:0/2277124621 wait complete. 2023-11-15T05:10:15.653 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:16.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:15 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:15.649926+0000 mon.smithi086 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.154:0/2277124621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:17.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:17.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:17.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:17 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:16.882229+0000 mgr.smithi086.hkfamo (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:18.217 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- 172.21.15.154:0/1813648416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 msgr2=0x7fa6cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:18.218 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 --2- 172.21.15.154:0/1813648416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc0f66c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa6bc009fb0 tx=0x7fa6bc009b30).stop 2023-11-15T05:10:18.218 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- 172.21.15.154:0/1813648416 shutdown_connections 2023-11-15T05:10:18.218 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 --2- 172.21.15.154:0/1813648416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc0f66c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:18.218 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- 172.21.15.154:0/1813648416 >> 172.21.15.154:0/1813648416 conn(0x7fa6cc0f1aa0 msgr2=0x7fa6cc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:18.218 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- 172.21.15.154:0/1813648416 shutdown_connections 2023-11-15T05:10:18.219 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- 172.21.15.154:0/1813648416 wait complete. 2023-11-15T05:10:18.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 Processor -- start 2023-11-15T05:10:18.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.217+0000 7fa6d317c700 1 -- start start 2023-11-15T05:10:18.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d317c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:18.221 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d317c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1069a0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d217a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d217a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:35208/0 (socket says 172.21.15.154:35208) 2023-11-15T05:10:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d217a700 1 -- 172.21.15.154:0/2499968252 learned_addr learned my addr 172.21.15.154:0/2499968252 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d217a700 1 -- 172.21.15.154:0/2499968252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc005040 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.218+0000 7fa6d217a700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa6bc013510 tx=0x7fa6bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.219+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6bc009320 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.219+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc106ba0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.219+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6bc013f00 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.219+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6bc014dc0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.219+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc107030 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.223 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6bc01d3d0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6c2ffd700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:18.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6bc041090 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc045ad0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6d1979700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:18.224 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.220+0000 7fa6d1979700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 0x7fa6b8032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa6c80096a0 tx=0x7fa6c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:18.225 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.224+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6bc012900 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.429 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.428+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6cc107ee0 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.429 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.429+0000 7fa6c2ffd700 1 -- 172.21.15.154:0/2499968252 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6bc02b020 con 0x7fa6cc0f62a0 2023-11-15T05:10:18.430 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:18.430 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:18.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 msgr2=0x7fa6b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:18.431 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 0x7fa6b8032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa6c80096a0 tx=0x7fa6c8006b40).stop 2023-11-15T05:10:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 msgr2=0x7fa6cc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa6bc013510 tx=0x7fa6bc00b040).stop 2023-11-15T05:10:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 shutdown_connections 2023-11-15T05:10:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7fa6b802ff60 0x7fa6b8032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 --2- 172.21.15.154:0/2499968252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6cc0f62a0 0x7fa6cc1064b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:18.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 >> 172.21.15.154:0/2499968252 conn(0x7fa6cc0f1aa0 msgr2=0x7fa6cc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:18.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 shutdown_connections 2023-11-15T05:10:18.433 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:18.430+0000 7fa6d317c700 1 -- 172.21.15.154:0/2499968252 wait complete. 2023-11-15T05:10:18.433 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:18.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:18 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:18.429865+0000 mon.smithi086 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.154:0/2499968252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:19.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:19 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:18.882535+0000 mgr.smithi086.hkfamo (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:19.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:19.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:20.960 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 -- 172.21.15.154:0/2442800241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 msgr2=0x7ff9f40f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:20.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 --2- 172.21.15.154:0/2442800241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f40f91b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7ff9e4009fb0 tx=0x7ff9e4009b30).stop 2023-11-15T05:10:20.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 -- 172.21.15.154:0/2442800241 shutdown_connections 2023-11-15T05:10:20.961 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 --2- 172.21.15.154:0/2442800241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f40f91b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:20.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 -- 172.21.15.154:0/2442800241 >> 172.21.15.154:0/2442800241 conn(0x7ff9f40f0b70 msgr2=0x7ff9f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:20.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 -- 172.21.15.154:0/2442800241 shutdown_connections 2023-11-15T05:10:20.962 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.957+0000 7ff9f989a700 1 -- 172.21.15.154:0/2442800241 wait complete. 2023-11-15T05:10:20.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f989a700 1 Processor -- start 2023-11-15T05:10:20.963 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f989a700 1 -- start start 2023-11-15T05:10:20.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f989a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:20.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f989a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4106be0 con 0x7ff9f40601c0 2023-11-15T05:10:20.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f2ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:20.964 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f2ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:35228/0 (socket says 172.21.15.154:35228) 2023-11-15T05:10:20.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.958+0000 7ff9f2ffd700 1 -- 172.21.15.154:0/1893686978 learned_addr learned my addr 172.21.15.154:0/1893686978 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:20.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9f2ffd700 1 -- 172.21.15.154:0/1893686978 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4005040 con 0x7ff9f40601c0 2023-11-15T05:10:20.965 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9f2ffd700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7ff9e4013510 tx=0x7ff9e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:20.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e4009320 con 0x7ff9f40601c0 2023-11-15T05:10:20.969 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f4106de0 con 0x7ff9f40601c0 2023-11-15T05:10:20.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9e4013f00 con 0x7ff9f40601c0 2023-11-15T05:10:20.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e4014dc0 con 0x7ff9f40601c0 2023-11-15T05:10:20.970 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.959+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f4107270 con 0x7ff9f40601c0 2023-11-15T05:10:20.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.960+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9e40095f0 con 0x7ff9f40601c0 2023-11-15T05:10:20.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.960+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f4045ad0 con 0x7ff9f40601c0 2023-11-15T05:10:20.971 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.960+0000 7ff9db7fe700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:20.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.960+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9e4041040 con 0x7ff9f40601c0 2023-11-15T05:10:20.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.960+0000 7ff9f27fc700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:20.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.961+0000 7ff9f27fc700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 0x7ff9dc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff9e80096a0 tx=0x7ff9e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:20.972 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:20.965+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9e4014420 con 0x7ff9f40601c0 2023-11-15T05:10:21.169 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.169+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9f4108120 con 0x7ff9f40601c0 2023-11-15T05:10:21.170 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.169+0000 7ff9db7fe700 1 -- 172.21.15.154:0/1893686978 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9e4010230 con 0x7ff9f40601c0 2023-11-15T05:10:21.171 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:21.171 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:21.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 msgr2=0x7ff9dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:21.172 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 0x7ff9dc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff9e80096a0 tx=0x7ff9e8006b40).stop 2023-11-15T05:10:21.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 msgr2=0x7ff9f41066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:21.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7ff9e4013510 tx=0x7ff9e400b040).stop 2023-11-15T05:10:21.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 shutdown_connections 2023-11-15T05:10:21.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:21.173 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 --2- 172.21.15.154:0/1893686978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f40601c0 0x7ff9f41066f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:21.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 >> 172.21.15.154:0/1893686978 conn(0x7ff9f40f0b70 msgr2=0x7ff9f40f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:21.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 shutdown_connections 2023-11-15T05:10:21.174 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:21.171+0000 7ff9f989a700 1 -- 172.21.15.154:0/1893686978 wait complete. 2023-11-15T05:10:21.174 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:21.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:20 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:20.882874+0000 mgr.smithi086.hkfamo (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:22.240 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:21 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:21.170570+0000 mon.smithi086 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.154:0/1893686978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:22.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:10:22.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:10:23.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:22 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: cluster 2023-11-15T05:10:22.883387+0000 mgr.smithi086.hkfamo (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:10:23.650 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 -- 172.21.15.154:0/656894639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 msgr2=0x7f1ebc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:23.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 --2- 172.21.15.154:0/656894639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f1eac009fb0 tx=0x7f1eac009b30).stop 2023-11-15T05:10:23.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 -- 172.21.15.154:0/656894639 shutdown_connections 2023-11-15T05:10:23.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 --2- 172.21.15.154:0/656894639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:23.651 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 -- 172.21.15.154:0/656894639 >> 172.21.15.154:0/656894639 conn(0x7f1ebc0f0b70 msgr2=0x7f1ebc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 -- 172.21.15.154:0/656894639 shutdown_connections 2023-11-15T05:10:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.648+0000 7f1ec4192700 1 -- 172.21.15.154:0/656894639 wait complete. 2023-11-15T05:10:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec4192700 1 Processor -- start 2023-11-15T05:10:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec4192700 1 -- start start 2023-11-15T05:10:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec4192700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec4192700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0fb820 con 0x7f1ebc0f5410 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec1f2e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec1f2e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.154:39400/0 (socket says 172.21.15.154:39400) 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec1f2e700 1 -- 172.21.15.154:0/3948308451 learned_addr learned my addr 172.21.15.154:0/3948308451 (peer_addr_for_me v2:172.21.15.154:0/0) 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.649+0000 7f1ec1f2e700 1 -- 172.21.15.154:0/3948308451 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac005040 con 0x7f1ebc0f5410 2023-11-15T05:10:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1ec1f2e700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1eac013510 tx=0x7f1eac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:23.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eac009320 con 0x7f1ebc0f5410 2023-11-15T05:10:23.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0fb9c0 con 0x7f1ebc0f5410 2023-11-15T05:10:23.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1eac013f00 con 0x7f1ebc0f5410 2023-11-15T05:10:23.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1eac014d90 con 0x7f1ebc0f5410 2023-11-15T05:10:23.654 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc0fbe50 con 0x7f1ebc0f5410 2023-11-15T05:10:23.655 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.651+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1eac0095f0 con 0x7f1ebc0f5410 2023-11-15T05:10:23.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.650+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc059100 con 0x7f1ebc0f5410 2023-11-15T05:10:23.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.651+0000 7f1eb27fc700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 0x7f1ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:10:23.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.651+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1eac041000 con 0x7f1ebc0f5410 2023-11-15T05:10:23.656 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.651+0000 7f1ec172d700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 0x7f1ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:10:23.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.651+0000 7f1ec172d700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 0x7f1ea8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1eb80096a0 tx=0x7f1eb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:10:23.657 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.655+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1eac014420 con 0x7f1ebc0f5410 2023-11-15T05:10:23.862 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.862+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ebc0fae60 con 0x7f1ebc0f5410 2023-11-15T05:10:23.863 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.862+0000 7f1eb27fc700 1 -- 172.21.15.154:0/3948308451 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1eac010230 con 0x7f1ebc0f5410 2023-11-15T05:10:23.863 INFO:teuthology.orchestra.run.smithi154.stdout: 2023-11-15T05:10:23.864 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"d8947496-8373-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:00:31.304747Z","created":"2023-11-15T05:00:31.304747Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:10:23.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 msgr2=0x7f1ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:23.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 0x7f1ea8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1eb80096a0 tx=0x7f1eb8006b40).stop 2023-11-15T05:10:23.865 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 msgr2=0x7f1ebc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:10:23.866 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f1eac013510 tx=0x7f1eac00b040).stop 2023-11-15T05:10:23.866 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 shutdown_connections 2023-11-15T05:10:23.866 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:6800/432331256,v1:172.21.15.86:6801/432331256] conn(0x7f1ea802ff60 0x7f1ea8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:23.866 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 --2- 172.21.15.154:0/3948308451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ebc0f5410 0x7f1ebc0fd3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:10:23.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.864+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 >> 172.21.15.154:0/3948308451 conn(0x7f1ebc0f0b70 msgr2=0x7f1ebc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:10:23.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.865+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 shutdown_connections 2023-11-15T05:10:23.867 INFO:teuthology.orchestra.run.smithi154.stderr:2023-11-15T05:10:23.865+0000 7f1ec4192700 1 -- 172.21.15.154:0/3948308451 wait complete. 2023-11-15T05:10:23.868 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2023-11-15T05:10:24.239 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:23 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: audit 2023-11-15T05:10:23.863642+0000 mon.smithi086 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.154:0/3948308451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:10:24.263 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T05:10:24.264 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T05:10:24.264 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:10:24.289 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:10:24.307 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T05:10:24.308 INFO:tasks.cephadm.mon.smithi086:Stopping mon.smithi086... 2023-11-15T05:10:24.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086 2023-11-15T05:10:24.642 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:24 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for d8947496-8373-11ee-8dbe-212e2dc638e7... 2023-11-15T05:10:24.642 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:10:24.419+0000 7f88a566a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T05:10:24.642 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Nov 15 05:10:24 smithi086 ceph-d8947496-8373-11ee-8dbe-212e2dc638e7-mon.smithi086[112320]: debug 2023-11-15T05:10:24.419+0000 7f88a566a700 -1 mon.smithi086@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T05:10:24.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi086.service' 2023-11-15T05:10:24.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T05:10:24.820 INFO:tasks.cephadm.mon.smithi086:Stopped mon.smithi086 2023-11-15T05:10:24.820 INFO:tasks.cephadm.mon.smithi154:Stopping mon.smithi154... 2023-11-15T05:10:24.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154 2023-11-15T05:10:24.854 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154.service: Unit ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154.service not loaded. 2023-11-15T05:10:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T05:10:24.856 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi154" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154' 2023-11-15T05:10:24.856 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T05:10:24.859 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/ubuntu@smithi086.front.sepia.ceph.com/crash 2023-11-15T05:10:24.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T05:10:24.899 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/ubuntu@smithi154.front.sepia.ceph.com/crash 2023-11-15T05:10:24.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T05:10:24.938 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T05:10:24.938 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2023-11-15T05:10:24.939 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T05:10:24.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T05:10:24.973 INFO:teuthology.orchestra.run.smithi086.stderr:grep: /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T05:10:24.975 INFO:tasks.cephadm:Compressing logs... 2023-11-15T05:10:24.976 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T05:10:25.018 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T05:10:25.044 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T05:10:25.044 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2023-11-15T05:10:25.044 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T05:10:25.045 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi086.log 2023-11-15T05:10:25.045 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi086.hkfamo.log 2023-11-15T05:10:25.046 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi086.log: /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi086.hkfamo.log: 88.4% -- replaced with /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T05:10:25.046 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T05:10:25.046 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2023-11-15T05:10:25.047 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T05:10:25.048 INFO:teuthology.orchestra.run.smithi086.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T05:10:25.049 INFO:teuthology.orchestra.run.smithi154.stderr: 2023-11-15T05:10:25.049 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m0.013s 2023-11-15T05:10:25.049 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m0.014s 2023-11-15T05:10:25.050 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.007s 2023-11-15T05:10:25.124 INFO:teuthology.orchestra.run.smithi086.stderr:gzip: /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi086.hkfamo.log: file size changed while zipping 2023-11-15T05:10:25.124 INFO:teuthology.orchestra.run.smithi086.stderr: 92.7% -- replaced with /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi086.hkfamo.log.gz 2023-11-15T05:10:25.192 INFO:teuthology.orchestra.run.smithi086.stderr: 90.6% -- replaced with /var/log/ceph/d8947496-8373-11ee-8dbe-212e2dc638e7/ceph-mon.smithi086.log.gz 2023-11-15T05:10:25.192 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-11-15T05:10:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.159s 2023-11-15T05:10:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.237s 2023-11-15T05:10:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.015s 2023-11-15T05:10:25.193 INFO:tasks.cephadm:Archiving logs... 2023-11-15T05:10:25.193 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/ubuntu@smithi086.front.sepia.ceph.com/log 2023-11-15T05:10:25.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T05:10:25.308 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/ubuntu@smithi154.front.sepia.ceph.com/log 2023-11-15T05:10:25.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T05:10:25.339 INFO:tasks.cephadm:Removing cluster... 2023-11-15T05:10:25.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 --force 2023-11-15T05:12:09.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d8947496-8373-11ee-8dbe-212e2dc638e7 --force 2023-11-15T05:12:10.302 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T05:12:10.303 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T05:12:10.323 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T05:12:10.341 INFO:tasks.cephadm:Teardown complete 2023-11-15T05:12:10.342 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154' 2023-11-15T05:12:10.422 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d97974a50a074cdbb8e6751beaddd370 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 5: 'sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154' 2023-11-15T05:12:10.428 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T05:12:10.442 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T05:12:10.455 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T05:12:10.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T05:12:10.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T05:12:10.563 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T05:12:10.575 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T05:12:10.576 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:12:10.579 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:12:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-11-15T05:12:10.597 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2023-11-15T05:12:10.598 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:12:10.598 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-11-15T05:12:10.598 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +192us[ +192us] +/- 77ms 2023-11-15T05:12:10.599 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -280us[ -254us] +/- 55ms 2023-11-15T05:12:10.599 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +266us[ +266us] +/- 65ms 2023-11-15T05:12:10.599 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:12:10.601 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:12:10.601 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2023-11-15T05:12:10.602 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +189us[ +209us] +/- 77ms 2023-11-15T05:12:10.602 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -283us[ -263us] +/- 55ms 2023-11-15T05:12:10.602 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +239us[ +260us] +/- 65ms 2023-11-15T05:12:10.603 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:12:10.604 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T05:12:10.614 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T05:12:10.615 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T05:12:10.626 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T05:12:10.728 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T05:12:10.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:12:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:12:10.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:12:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:12:10.885 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T05:12:10.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T05:12:10.904 INFO:teuthology.task.internal:Duration was 1396.144529 seconds 2023-11-15T05:12:10.905 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T05:12:10.915 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T05:12:10.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T05:12:10.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T05:12:10.963 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:12:10.971 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:12:11.249 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T05:12:11.249 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2023-11-15T05:12:11.250 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T05:12:11.273 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2023-11-15T05:12:11.274 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T05:12:11.295 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T05:12:11.296 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T05:12:11.316 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T05:12:11.381 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T05:12:11.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T05:12:11.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T05:12:11.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T05:12:11.509 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T05:12:11.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T05:12:11.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T05:12:11.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T05:12:11.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:12:11.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:12:11.605 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2023-11-15T05:12:11.633 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2023-11-15T05:12:11.653 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:12:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:12:11.703 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T05:12:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:12:11.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T05:12:11.731 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T05:12:11.732 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/smithi086 2023-11-15T05:12:11.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T05:12:11.784 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425/remote/smithi154 2023-11-15T05:12:11.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T05:12:11.825 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T05:12:11.826 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T05:12:11.829 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T05:12:11.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T05:12:11.889 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T05:12:11.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T05:12:11.900 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T05:12:11.900 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T05:12:11.903 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T05:12:11.918 INFO:teuthology.orchestra.run.smithi086.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 05:12 /home/ubuntu/cephtest 2023-11-15T05:12:11.935 INFO:teuthology.orchestra.run.smithi154.stdout: 525836 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 05:12 /home/ubuntu/cephtest 2023-11-15T05:12:11.936 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T05:12:11.948 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T05:12:11.973 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T05:12:11.999 DEBUG:teuthology.nuke:shortname: smithi086 2023-11-15T05:12:12.000 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T05:12:12.024 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:40:59.062198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmNWvIMYMBqczIfLppAztcZA0wBWzxz1SL+1QQMOdvHgZKGWzS4nzHF+OM/LI44DKYrFMadiP6k+Em4HBFUAG0='} 2023-11-15T05:12:12.028 DEBUG:teuthology.nuke:shortname: smithi154 2023-11-15T05:12:12.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T05:12:12.055 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458425', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 04:40:59.064060', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcj3y44ysMXUmndcxZQ8cxB+MnBdbRyL1lJw1EbK1de/q/daTKUJBoA5iDZMVZA2nrRt7+eyUqkASeKsHiNICE='} 2023-11-15T05:12:12.086 INFO:teuthology.orchestra.console.smithi086:Power off 2023-11-15T05:12:12.087 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T05:12:12.110 INFO:teuthology.orchestra.console.smithi154:Power off 2023-11-15T05:12:12.111 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T05:12:12.121 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2023-11-15T05:12:12.122 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:12.137 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2023-11-15T05:12:12.138 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:12.149 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-11-15T05:12:12.164 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-11-15T05:12:16.151 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:16.165 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:16.263 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-11-15T05:12:16.275 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2023-11-15T05:12:20.266 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:20.276 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:20.379 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-11-15T05:12:20.387 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2023-11-15T05:12:20.488 INFO:teuthology.orchestra.console.smithi154:Power off completed 2023-11-15T05:12:24.384 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T05:12:24.496 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2023-11-15T05:12:24.598 INFO:teuthology.orchestra.console.smithi086:Power off completed 2023-11-15T05:12:24.754 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1396.1445288658142 failure_reason: 'Command failed on smithi154 with status 5: ''sudo systemctl stop ceph-d8947496-8373-11ee-8dbe-212e2dc638e7@mon.smithi154''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d97974a50a074cdbb8e6751beaddd370 status: fail success: false 2023-11-15T05:12:24.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T05:12:24.864 INFO:teuthology.run:FAIL