2023-11-15T05:37:06.614 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T05:37:06.615 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/orig.config.yaml 2023-11-15T05:37:06.670 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T05:37:06.764 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458530' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOXwiuJfu0mmNPAy+Zd1ksMS5wpRkhqhKpdWlFA/ku+vv509YqaBYDr8EuJECVM8hJLam7DXA2h6UBsHaeUO1E= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICrjbq4dUH/QpLH8b3n08i+dpWeh1eYa74dN94LX7sFrf7N/CUHLvmM+nk5Jm/D3t7bTfjpbR6BDNHLpP6AsD0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T05:37:06.764 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T05:37:06.765 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T05:37:06.766 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T05:37:06.768 INFO:teuthology.task.internal:Checking packages... 2023-11-15T05:37:06.795 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T05:37:06.796 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T05:37:06.796 INFO:teuthology.packaging:ref: None 2023-11-15T05:37:06.797 INFO:teuthology.packaging:tag: None 2023-11-15T05:37:06.797 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T05:37:06.798 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:37:06.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T05:37:06.937 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T05:37:06.937 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T05:37:06.944 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T05:37:06.944 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T05:37:06.967 INFO:teuthology.task.internal:Saving configuration 2023-11-15T05:37:06.989 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T05:37:07.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T05:37:07.030 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 05:31:46.767699', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOXwiuJfu0mmNPAy+Zd1ksMS5wpRkhqhKpdWlFA/ku+vv509YqaBYDr8EuJECVM8hJLam7DXA2h6UBsHaeUO1E='} 2023-11-15T05:37:07.055 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 05:31:46.766042', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICrjbq4dUH/QpLH8b3n08i+dpWeh1eYa74dN94LX7sFrf7N/CUHLvmM+nk5Jm/D3t7bTfjpbR6BDNHLpP6AsD0='} 2023-11-15T05:37:07.056 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T05:37:07.062 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T05:37:07.062 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T05:37:07.063 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T05:37:07.136 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9fced5bb80>, signals=[15]) 2023-11-15T05:37:07.137 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T05:37:07.144 INFO:teuthology.task.internal:Opening connections... 2023-11-15T05:37:07.145 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-11-15T05:37:07.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:37:07.257 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T05:37:07.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:37:07.327 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T05:37:07.336 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-11-15T05:37:07.361 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-11-15T05:37:07.361 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-11-15T05:37:07.423 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2023-11-15T05:37:07.423 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8" 2023-11-15T05:37:07.423 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2023-11-15T05:37:07.423 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2023-11-15T05:37:07.423 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8" 2023-11-15T05:37:07.424 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-11-15T05:37:07.424 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T05:37:07.424 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-11-15T05:37:07.424 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T05:37:07.425 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2023-11-15T05:37:07.425 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T05:37:07.425 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T05:37:07.425 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T05:37:07.426 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-11-15T05:37:07.449 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2023-11-15T05:37:07.472 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2023-11-15T05:37:07.473 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2023-11-15T05:37:07.535 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2023-11-15T05:37:07.535 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2023-11-15T05:37:07.535 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2023-11-15T05:37:07.535 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2023-11-15T05:37:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2023-11-15T05:37:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2023-11-15T05:37:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T05:37:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2023-11-15T05:37:07.537 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T05:37:07.537 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2023-11-15T05:37:07.537 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T05:37:07.537 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T05:37:07.537 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T05:37:07.538 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2023-11-15T05:37:07.557 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T05:37:07.568 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T05:37:07.578 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T05:37:07.578 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-11-15T05:37:07.582 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2023-11-15T05:37:07.603 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T05:37:07.611 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T05:37:07.612 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-11-15T05:37:07.644 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2023-11-15T05:37:07.686 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T05:37:07.745 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T05:37:07.763 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T05:37:07.764 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T05:37:07.764 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T05:37:07.765 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T05:37:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:37:07.786 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-11-15T05:37:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:37:07.788 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-11-15T05:37:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2023-11-15T05:37:07.849 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2023-11-15T05:37:07.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2023-11-15T05:37:07.850 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-489.el8.x86_64 2023-11-15T05:37:07.851 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-489.el8.x86_64 2023-11-15T05:37:07.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-11-15T05:37:18.109 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 34 MB 00:07 2023-11-15T05:37:19.277 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - AppStream 3.9 MB/s | 34 MB 00:08 2023-11-15T05:37:27.092 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 53 MB 00:02 2023-11-15T05:37:28.318 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 53 MB 00:02 2023-11-15T05:37:35.750 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 16 kB/s | 18 kB 00:01 2023-11-15T05:37:36.533 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-11-15T05:37:36.802 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-11-15T05:37:37.259 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras 20 kB/s | 18 kB 00:00 2023-11-15T05:37:37.556 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-11-15T05:37:38.202 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras common packages 9.3 kB/s | 6.9 kB 00:00 2023-11-15T05:37:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-11-15T05:37:39.706 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-11-15T05:37:41.388 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-11-15T05:37:43.746 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-11-15T05:37:45.029 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T05:37:45.029 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T05:37:45.291 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:37:45.291 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:37:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:37:45.292 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:37:45.292 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:37:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T05:37:45.292 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:37:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T05:37:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T05:37:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:37:45.293 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:37:45.293 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:37:45.294 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2023-11-15T05:37:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:37:45.295 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 89 M 2023-11-15T05:37:45.295 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2023-11-15T05:37:45.295 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:37:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.5 MB/s | 10 MB 00:01 2023-11-15T05:37:47.537 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T05:37:47.537 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T05:37:47.639 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2023-11-15T05:37:47.843 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:37:47.844 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:37:47.844 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:37:47.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:37:47.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:37:47.845 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T05:37:47.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:37:47.845 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T05:37:47.846 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T05:37:47.846 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:37:47.846 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:37:47.846 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:37:47.846 INFO:teuthology.orchestra.run.smithi003.stdout:Install 3 Packages 2023-11-15T05:37:47.847 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:37:47.847 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 89 M 2023-11-15T05:37:47.848 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 96 M 2023-11-15T05:37:47.848 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:37:48.206 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-11-15T05:37:48.207 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:37:48.207 INFO:teuthology.orchestra.run.smithi137.stdout:Total 31 MB/s | 89 MB 00:02 2023-11-15T05:37:48.891 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:37:49.105 INFO:teuthology.orchestra.run.smithi003.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.2 MB/s | 10 MB 00:01 2023-11-15T05:37:49.308 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:37:49.308 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:37:50.289 INFO:teuthology.orchestra.run.smithi003.stdout:(2/3): kernel-core-4.18.0-521.el8.x86_64.rpm 18 MB/s | 43 MB 00:02 2023-11-15T05:37:50.557 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:37:50.595 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:37:50.639 INFO:teuthology.orchestra.run.smithi003.stdout:(3/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2023-11-15T05:37:50.639 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:37:50.640 INFO:teuthology.orchestra.run.smithi003.stdout:Total 32 MB/s | 89 MB 00:02 2023-11-15T05:37:51.349 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:37:51.796 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:37:51.796 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:37:53.095 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:37:53.115 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:37:53.352 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:37:53.883 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:37:56.076 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:37:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:37:57.537 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:37:57.915 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:00.391 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:38:00.762 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:01.628 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:04.436 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:04.712 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:23.939 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:23.939 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:38:23.940 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:24.777 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:24.777 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:38:24.777 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:38:24.777 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T05:38:24.778 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T05:38:24.778 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:38:24.778 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:38:24.955 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T05:38:25.189 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T05:38:25.190 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T05:38:25.190 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T05:38:25.190 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T05:38:25.190 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T05:38:25.191 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T05:38:25.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T05:38:25.191 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T05:38:25.401 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T05:38:25.402 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T05:38:25.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2023-11-15T05:38:25.436 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2023-11-15T05:38:25.436 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2023-11-15T05:38:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2023-11-15T05:38:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2023-11-15T05:38:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2023-11-15T05:38:25.437 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T05:38:25.438 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2023-11-15T05:38:25.438 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2023-11-15T05:38:25.438 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2023-11-15T05:38:25.438 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T05:38:25.438 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T05:38:25.439 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T05:38:25.439 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T05:38:25.439 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2023-11-15T05:38:25.440 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2023-11-15T05:38:25.440 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2023-11-15T05:38:25.440 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T05:38:25.440 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2023-11-15T05:38:25.441 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2023-11-15T05:38:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:38:25.441 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T05:38:25.441 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T05:38:25.441 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T05:38:25.442 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2023-11-15T05:38:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:38:25.442 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T05:38:25.444 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T05:38:25.444 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-521.el8.x86_64 2023-11-15T05:38:25.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T05:38:27.143 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2023-11-15T05:38:28.160 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:28.537 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T05:38:28.538 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T05:38:29.301 INFO:teuthology.orchestra.run.smithi137.stderr:done 2023-11-15T05:38:29.302 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2023-11-15T05:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.wuIIwfnuUj 2023-11-15T05:38:29.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wuIIwfnuUj 2023-11-15T05:38:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T05:38:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:38:29.370 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:38:29.370 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T05:38:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T05:38:29.371 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:38:29.371 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:38:29.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.wuIIwfnuUj 2023-11-15T05:38:29.511 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T05:38:29.534 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.wuIIwfnuUj is 6KB 2023-11-15T05:38:29.584 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.wuIIwfnuUj 2023-11-15T05:38:29.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T05:38:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T05:38:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T05:38:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T05:38:29.677 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T05:38:29.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T05:38:29.782 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T05:38:29.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T05:38:29.783 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T05:38:29.783 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T05:38:29.783 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T05:38:29.783 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T05:38:29.784 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T05:38:29.784 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T05:38:29.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2023-11-15T05:38:30.002 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T05:38:30.002 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T05:38:30.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-11-15T05:38:30.037 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-11-15T05:38:30.037 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-11-15T05:38:30.038 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.02 2023-11-15T05:38:30.038 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 129.el8 2023-11-15T05:38:30.038 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-11-15T05:38:30.038 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T05:38:30.039 INFO:teuthology.orchestra.run.smithi003.stdout:Group : System Environment/Base 2023-11-15T05:38:30.039 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 9265586 2023-11-15T05:38:30.039 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-11-15T05:38:30.039 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T05:38:30.039 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T05:38:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T05:38:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T05:38:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:Relocations : (not relocatable) 2023-11-15T05:38:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : CentOS BuildSystem 2023-11-15T05:38:30.041 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2023-11-15T05:38:30.041 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T05:38:30.041 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-11-15T05:38:30.041 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-11-15T05:38:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:38:30.042 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T05:38:30.042 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T05:38:30.042 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T05:38:30.043 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-11-15T05:38:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:38:30.043 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T05:38:30.044 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T05:38:30.044 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 4.18.0-521.el8.x86_64 2023-11-15T05:38:30.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T05:38:31.685 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-11-15T05:38:33.530 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-11-15T05:38:33.532 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-11-15T05:38:33.553 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.8oVOSFBV6A 2023-11-15T05:38:33.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8oVOSFBV6A 2023-11-15T05:38:33.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.8oVOSFBV6A 2023-11-15T05:38:33.764 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.8oVOSFBV6A is 6KB 2023-11-15T05:38:33.813 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.8oVOSFBV6A 2023-11-15T05:38:33.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T05:38:33.902 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T05:38:33.902 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T05:38:33.902 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T05:38:33.902 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T05:38:33.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T05:38:34.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-11-15T05:38:59.890 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T05:38:59.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:38:59.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:39:04.086 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T05:39:04.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:39:04.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:39:18.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2023-11-15T05:39:22.619 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-11-15T05:39:27.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:39:27.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:39:31.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:39:31.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:39:34.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-11-15T05:39:46.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:39:46.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:40:18.524 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-11-15T05:40:18.789 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:40:18.789 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T05:40:18.789 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-11-15T05:40:18.810 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-521.el8.x86_64 2023-11-15T05:40:18.810 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T05:40:18.810 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T05:40:18.811 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T05:40:19.812 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T05:40:19.812 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T05:40:19.939 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2023-11-15T05:40:19.955 DEBUG:teuthology.parallel:result is None 2023-11-15T05:40:27.275 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T05:40:39.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:40:39.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:40:39.542 DEBUG:teuthology.orchestra.run.smithi137:> true 2023-11-15T05:40:39.844 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:40:39.844 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T05:40:39.845 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2023-11-15T05:40:39.864 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-521.el8.x86_64 2023-11-15T05:40:39.865 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T05:40:39.865 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T05:40:39.865 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T05:40:40.866 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T05:40:40.867 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T05:40:40.964 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2023-11-15T05:40:40.985 DEBUG:teuthology.parallel:result is None 2023-11-15T05:40:40.985 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T05:40:40.993 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T05:40:40.993 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T05:40:40.996 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T05:40:41.017 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T05:40:41.026 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T05:40:41.036 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T05:40:41.036 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T05:40:41.060 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T05:40:41.100 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T05:40:41.108 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T05:40:41.109 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T05:40:41.138 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T05:40:41.177 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T05:40:41.180 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T05:40:41.187 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T05:40:41.192 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T05:40:41.194 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T05:40:41.201 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T05:40:41.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T05:40:41.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T05:40:41.258 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T05:40:41.267 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T05:40:41.268 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T05:40:41.277 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T05:40:41.317 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T05:40:41.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T05:40:41.453 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T05:40:41.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T05:40:41.552 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:40:41.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T05:40:41.626 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T05:40:41.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T05:40:41.723 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T05:40:41.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T05:40:41.853 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:40:41.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T05:40:41.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-11-15T05:40:41.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2023-11-15T05:40:42.005 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:40:42.018 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T05:40:42.447 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T05:40:42.456 INFO:teuthology.task.internal:Starting timer... 2023-11-15T05:40:42.456 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T05:40:42.472 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T05:40:42.483 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T05:40:42.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-11-15T05:40:42.615 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-11-15T05:40:42.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2023-11-15T05:40:42.683 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2023-11-15T05:40:42.685 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T05:40:42.685 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-11-15T05:40:42.722 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-11-15T05:40:42.722 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2023-11-15T05:40:42.744 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2023-11-15T05:40:42.745 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2023-11-15T05:40:42.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:40:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:40:42.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T05:40:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T05:40:42.825 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T05:40:42.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-11-15T05:40:42.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2023-11-15T05:40:42.899 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T05:40:42.908 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T05:40:42.909 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T05:40:42.925 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T05:40:42.926 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T05:48:59.726 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2023-11-15T05:48:59.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:48:59.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:48:59.799 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-11-15T05:48:59.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-15T05:48:59.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:48:59.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T05:48:59.943 DEBUG:teuthology.orchestra.run.smithi137:> true 2023-11-15T05:49:00.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-11-15T05:49:00.017 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T05:49:00.029 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T05:49:00.030 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T05:49:00.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:49:00.034 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T05:49:00.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T05:49:00.071 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T05:49:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T05:49:00.088 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T05:49:00.102 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T05:49:00.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-11-15T05:49:00.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2023-11-15T05:49:00.171 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-11-15T05:49:00.172 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2023-11-15T05:49:00.185 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T05:49:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T05:49:00.198 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T05:49:00.203 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T05:49:00.271 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-11-15T05:49:00.272 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2023-11-15T05:49:00.275 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:49:00.275 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-11-15T05:49:00.275 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.275 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.276 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.276 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.278 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T05:49:00.278 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2023-11-15T05:49:00.278 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.278 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.279 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.279 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T05:49:00.279 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T05:49:00.289 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T05:49:00.290 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-11-15T05:49:00.291 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T05:49:00.291 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T05:49:00.292 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T05:49:01.199 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 05:46:34 AM UTC. 2023-11-15T05:49:01.340 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Wed 15 Nov 2023 05:47:18 AM UTC. 2023-11-15T05:49:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:49:03.297 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:49:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:Resetting modules: 2023-11-15T05:49:03.299 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools 2023-11-15T05:49:03.299 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:03.300 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:49:03.300 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:03.300 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:03.550 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:49:03.698 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:49:03.699 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:49:03.700 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:03.700 INFO:teuthology.orchestra.run.smithi003.stdout:Resetting modules: 2023-11-15T05:49:03.700 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools 2023-11-15T05:49:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:03.701 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:49:03.701 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:03.701 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:03.971 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:49:04.518 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 05:46:34 AM UTC. 2023-11-15T05:49:05.050 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 Nov 2023 05:47:18 AM UTC. 2023-11-15T05:49:06.628 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:49:06.806 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:06.806 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:49:06.806 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:06.807 INFO:teuthology.orchestra.run.smithi137.stdout:Installing group/module packages: 2023-11-15T05:49:06.807 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T05:49:06.807 INFO:teuthology.orchestra.run.smithi137.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T05:49:06.807 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T05:49:06.807 INFO:teuthology.orchestra.run.smithi137.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T05:49:06.808 INFO:teuthology.orchestra.run.smithi137.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T05:49:06.808 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T05:49:06.808 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T05:49:06.808 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T05:49:06.808 INFO:teuthology.orchestra.run.smithi137.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T05:49:06.809 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:49:06.809 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T05:49:06.809 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T05:49:06.809 INFO:teuthology.orchestra.run.smithi137.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T05:49:06.809 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2023-11-15T05:49:06.810 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T05:49:06.810 INFO:teuthology.orchestra.run.smithi137.stdout:Installing module profiles: 2023-11-15T05:49:06.810 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools/common 2023-11-15T05:49:06.810 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:06.810 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling module streams: 2023-11-15T05:49:06.811 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools rhel8 2023-11-15T05:49:06.811 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:06.811 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:49:06.812 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:06.813 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2023-11-15T05:49:06.813 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:06.814 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2023-11-15T05:49:06.814 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 89 M 2023-11-15T05:49:06.814 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:49:07.135 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 5.2 MB/s | 750 kB 00:00 2023-11-15T05:49:07.169 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 5.8 MB/s | 1.0 MB 00:00 2023-11-15T05:49:07.270 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 283 kB/s | 38 kB 00:00 2023-11-15T05:49:07.286 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.0 MB/s | 241 kB 00:00 2023-11-15T05:49:07.337 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.5 MB/s | 178 kB 00:00 2023-11-15T05:49:07.381 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:49:07.396 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2023-11-15T05:49:07.471 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 20 MB/s | 3.9 MB 00:00 2023-11-15T05:49:07.496 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 936 kB/s | 94 kB 00:00 2023-11-15T05:49:07.580 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:07.581 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:49:07.581 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:07.581 INFO:teuthology.orchestra.run.smithi003.stdout:Installing group/module packages: 2023-11-15T05:49:07.581 INFO:teuthology.orchestra.run.smithi003.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T05:49:07.581 INFO:teuthology.orchestra.run.smithi003.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T05:49:07.582 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T05:49:07.582 INFO:teuthology.orchestra.run.smithi003.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T05:49:07.582 INFO:teuthology.orchestra.run.smithi003.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T05:49:07.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T05:49:07.582 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T05:49:07.583 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T05:49:07.583 INFO:teuthology.orchestra.run.smithi003.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T05:49:07.583 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:49:07.583 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T05:49:07.583 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T05:49:07.584 INFO:teuthology.orchestra.run.smithi003.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T05:49:07.584 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2023-11-15T05:49:07.584 INFO:teuthology.orchestra.run.smithi003.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T05:49:07.584 INFO:teuthology.orchestra.run.smithi003.stdout:Installing module profiles: 2023-11-15T05:49:07.584 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools/common 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools rhel8 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:49:07.585 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:07.586 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-11-15T05:49:07.586 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:07.587 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 27 M 2023-11-15T05:49:07.587 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 89 M 2023-11-15T05:49:07.587 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:49:07.696 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 14 MB/s | 9.6 MB 00:00 2023-11-15T05:49:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 9.5 MB/s | 2.5 MB 00:00 2023-11-15T05:49:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 629 kB/s | 53 kB 00:00 2023-11-15T05:49:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 1.2 MB/s | 41 kB 00:00 2023-11-15T05:49:07.906 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 19 MB/s | 8.1 MB 00:00 2023-11-15T05:49:07.906 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:49:07.907 INFO:teuthology.orchestra.run.smithi137.stdout:Total 24 MB/s | 27 MB 00:01 2023-11-15T05:49:07.921 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 5.3 MB/s | 750 kB 00:00 2023-11-15T05:49:07.955 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 5.8 MB/s | 1.0 MB 00:00 2023-11-15T05:49:07.972 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 753 kB/s | 38 kB 00:00 2023-11-15T05:49:07.997 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 5.6 MB/s | 241 kB 00:00 2023-11-15T05:49:08.062 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.7 MB/s | 178 kB 00:00 2023-11-15T05:49:08.106 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 560 kB/s | 25 kB 00:00 2023-11-15T05:49:08.148 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 2.2 MB/s | 94 kB 00:00 2023-11-15T05:49:08.174 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:49:08.195 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:49:08.196 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:49:08.215 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 16 MB/s | 3.9 MB 00:00 2023-11-15T05:49:08.357 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 18 MB/s | 2.5 MB 00:00 2023-11-15T05:49:08.399 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.2 MB/s | 53 kB 00:00 2023-11-15T05:49:08.441 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 983 kB/s | 41 kB 00:00 2023-11-15T05:49:08.445 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:49:08.448 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:49:08.591 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 18 MB/s | 8.1 MB 00:00 2023-11-15T05:49:08.816 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 9.3 MB/s | 9.6 MB 00:01 2023-11-15T05:49:08.817 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:49:08.817 INFO:teuthology.orchestra.run.smithi003.stdout:Total 22 MB/s | 27 MB 00:01 2023-11-15T05:49:09.108 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:49:09.132 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:49:09.133 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:49:09.158 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:49:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T05:49:09.429 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:49:09.432 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:49:09.506 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T05:49:09.704 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T05:49:09.913 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T05:49:10.190 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:49:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T05:49:10.444 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T05:49:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T05:49:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T05:49:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T05:49:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T05:49:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T05:49:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T05:49:11.814 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T05:49:12.090 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T05:49:12.500 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T05:49:12.740 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T05:49:13.654 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T05:49:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T05:49:13.844 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T05:49:13.951 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T05:49:14.009 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T05:49:14.211 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T05:49:14.441 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T05:49:14.441 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T05:49:14.441 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T05:49:14.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T05:49:14.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T05:49:14.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T05:49:14.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T05:49:14.442 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T05:49:14.443 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T05:49:14.443 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T05:49:14.443 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T05:49:14.443 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T05:49:14.444 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T05:49:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T05:49:15.254 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T05:49:15.254 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:15.255 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:49:15.255 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.255 INFO:teuthology.orchestra.run.smithi137.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.255 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:15.255 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T05:49:15.256 INFO:teuthology.orchestra.run.smithi137.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.256 INFO:teuthology.orchestra.run.smithi137.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.256 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:15.256 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T05:49:15.256 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T05:49:15.257 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.257 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:15.257 INFO:teuthology.orchestra.run.smithi137.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:15.257 INFO:teuthology.orchestra.run.smithi137.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T05:49:15.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:15.258 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:49:15.535 DEBUG:teuthology.parallel:result is None 2023-11-15T05:49:15.594 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T05:49:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T05:49:15.995 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T05:49:16.503 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T05:49:16.503 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T05:49:16.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T05:49:16.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T05:49:16.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T05:49:16.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T05:49:16.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T05:49:16.505 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T05:49:16.505 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T05:49:16.505 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T05:49:16.505 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T05:49:16.505 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T05:49:16.506 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T05:49:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T05:49:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:17.446 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:49:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T05:49:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T05:49:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T05:49:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T05:49:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T05:49:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T05:49:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:17.449 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:49:17.748 DEBUG:teuthology.parallel:result is None 2023-11-15T05:49:17.748 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T05:49:17.760 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T05:49:17.761 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-11-15T05:49:17.761 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T05:49:17.762 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T05:49:17.762 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T05:49:17.919 DEBUG:teuthology.parallel:result is None 2023-11-15T05:49:17.968 DEBUG:teuthology.parallel:result is None 2023-11-15T05:49:17.969 INFO:teuthology.run_tasks:Running task install... 2023-11-15T05:49:17.978 DEBUG:teuthology.task.install:project ceph 2023-11-15T05:49:17.978 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T05:49:17.979 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T05:49:17.979 INFO:teuthology.task.install:Using flavor: default 2023-11-15T05:49:17.985 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T05:49:17.985 INFO:teuthology.task.install:extra packages: [] 2023-11-15T05:49:17.986 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T05:49:17.986 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:49:17.987 INFO:teuthology.packaging:ref: None 2023-11-15T05:49:17.987 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:49:17.987 INFO:teuthology.packaging:branch: None 2023-11-15T05:49:17.987 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:49:18.192 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:49:18.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:49:18.196 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T05:49:18.196 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:49:18.196 INFO:teuthology.packaging:ref: None 2023-11-15T05:49:18.197 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:49:18.197 INFO:teuthology.packaging:branch: None 2023-11-15T05:49:18.197 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:49:18.406 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:49:18.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:49:18.550 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T05:49:18.551 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T05:49:18.568 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T05:49:18.568 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T05:49:18.626 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T05:49:18.626 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:49:18.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T05:49:18.629 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T05:49:18.629 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:49:18.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T05:49:18.662 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T05:49:18.663 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:49:18.663 INFO:teuthology.packaging:ref: None 2023-11-15T05:49:18.663 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:49:18.664 INFO:teuthology.packaging:branch: None 2023-11-15T05:49:18.664 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:49:18.664 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T05:49:18.667 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T05:49:18.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:49:18.668 INFO:teuthology.packaging:ref: None 2023-11-15T05:49:18.668 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:49:18.668 INFO:teuthology.packaging:branch: None 2023-11-15T05:49:18.668 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:49:18.668 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T05:49:18.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T05:49:18.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T05:49:18.815 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T05:49:18.830 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T05:49:18.864 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-11-15T05:49:18.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-11-15T05:49:18.891 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2023-11-15T05:49:18.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2023-11-15T05:49:19.171 INFO:teuthology.orchestra.run.smithi137.stdout:76 files removed 2023-11-15T05:49:19.172 INFO:teuthology.orchestra.run.smithi003.stdout:76 files removed 2023-11-15T05:49:19.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2023-11-15T05:49:19.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2023-11-15T05:49:19.700 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 786 kB/s | 74 kB 00:00 2023-11-15T05:49:19.759 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 972 kB/s | 74 kB 00:00 2023-11-15T05:49:19.818 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-11-15T05:49:19.884 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 183 kB/s | 15 kB 00:00 2023-11-15T05:49:19.918 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-11-15T05:49:20.030 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 15 kB/s | 1.7 kB 00:00 2023-11-15T05:49:27.193 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - AppStream 4.8 MB/s | 34 MB 00:07 2023-11-15T05:49:27.235 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 4.7 MB/s | 34 MB 00:07 2023-11-15T05:49:35.560 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 53 MB 00:02 2023-11-15T05:49:36.401 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-15T05:49:44.501 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 13 kB/s | 18 kB 00:01 2023-11-15T05:49:45.436 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 9.2 kB/s | 6.9 kB 00:00 2023-11-15T05:49:46.026 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras 17 kB/s | 18 kB 00:01 2023-11-15T05:49:46.236 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.6 MB 00:00 2023-11-15T05:49:46.877 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 6.9 kB 00:00 2023-11-15T05:49:47.636 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - PowerTools 11 MB/s | 6.6 MB 00:00 2023-11-15T05:49:47.671 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-15T05:49:48.849 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2023-11-15T05:49:49.271 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T05:49:50.271 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-11-15T05:49:52.797 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 368 kB/s | 24 kB 00:00 2023-11-15T05:49:54.680 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 601 kB/s | 24 kB 00:00 2023-11-15T05:49:57.353 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:49:57.355 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:57.356 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:49:57.356 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:57.356 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:49:57.356 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T05:49:57.357 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2023-11-15T05:49:57.357 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T05:49:57.357 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T05:49:57.357 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:49:57.357 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T05:49:57.358 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T05:49:57.358 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T05:49:57.358 INFO:teuthology.orchestra.run.smithi137.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T05:49:57.358 INFO:teuthology.orchestra.run.smithi137.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T05:49:57.358 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T05:49:57.359 INFO:teuthology.orchestra.run.smithi137.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T05:49:57.359 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T05:49:57.359 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T05:49:57.359 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T05:49:57.359 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T05:49:57.360 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T05:49:57.360 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T05:49:57.360 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T05:49:57.360 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T05:49:57.360 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T05:49:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T05:49:57.362 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T05:49:57.362 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T05:49:57.362 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:57.362 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:49:57.362 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:49:57.363 INFO:teuthology.orchestra.run.smithi137.stdout:Install 25 Packages 2023-11-15T05:49:57.363 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2023-11-15T05:49:57.363 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:49:57.364 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 59 M 2023-11-15T05:49:57.364 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:49:58.090 INFO:teuthology.orchestra.run.smithi137.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2023-11-15T05:49:58.107 INFO:teuthology.orchestra.run.smithi137.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-11-15T05:49:58.149 INFO:teuthology.orchestra.run.smithi137.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T05:49:58.191 INFO:teuthology.orchestra.run.smithi137.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T05:49:58.316 INFO:teuthology.orchestra.run.smithi137.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 33 MB/s | 4.1 MB 00:00 2023-11-15T05:49:58.342 INFO:teuthology.orchestra.run.smithi137.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2023-11-15T05:49:58.367 INFO:teuthology.orchestra.run.smithi137.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T05:49:58.392 INFO:teuthology.orchestra.run.smithi137.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-15T05:49:58.426 INFO:teuthology.orchestra.run.smithi137.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T05:49:58.451 INFO:teuthology.orchestra.run.smithi137.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2023-11-15T05:49:58.477 INFO:teuthology.orchestra.run.smithi137.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2023-11-15T05:49:58.644 INFO:teuthology.orchestra.run.smithi137.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2023-11-15T05:49:58.669 INFO:teuthology.orchestra.run.smithi137.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 756 kB/s | 145 kB 00:00 2023-11-15T05:49:58.719 INFO:teuthology.orchestra.run.smithi137.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-11-15T05:49:58.961 INFO:teuthology.orchestra.run.smithi137.stdout:(15/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-11-15T05:49:58.987 INFO:teuthology.orchestra.run.smithi137.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 164 kB/s | 44 kB 00:00 2023-11-15T05:49:59.045 INFO:teuthology.orchestra.run.smithi137.stdout:(17/27): librdkafka-0.11.4-3.el8.x86_64.rpm 882 kB/s | 354 kB 00:00 2023-11-15T05:49:59.312 INFO:teuthology.orchestra.run.smithi137.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 120 kB/s | 42 kB 00:00 2023-11-15T05:49:59.329 INFO:teuthology.orchestra.run.smithi137.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 137 kB/s | 47 kB 00:00 2023-11-15T05:49:59.455 INFO:teuthology.orchestra.run.smithi137.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 196 kB/s | 80 kB 00:00 2023-11-15T05:49:59.834 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:49:59.838 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:59.838 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:49:59.838 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:59.838 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:49:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T05:49:59.839 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-11-15T05:49:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T05:49:59.839 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T05:49:59.839 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:49:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T05:49:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T05:49:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T05:49:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T05:49:59.840 INFO:teuthology.orchestra.run.smithi003.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T05:49:59.841 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T05:49:59.841 INFO:teuthology.orchestra.run.smithi003.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T05:49:59.841 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T05:49:59.841 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T05:49:59.841 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T05:49:59.842 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T05:49:59.842 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T05:49:59.842 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T05:49:59.842 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T05:49:59.842 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T05:49:59.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T05:49:59.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T05:49:59.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T05:49:59.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T05:49:59.844 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:59.844 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:49:59.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:49:59.845 INFO:teuthology.orchestra.run.smithi003.stdout:Install 25 Packages 2023-11-15T05:49:59.845 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2023-11-15T05:49:59.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:49:59.846 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 59 M 2023-11-15T05:49:59.846 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:49:59.988 INFO:teuthology.orchestra.run.smithi137.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 172 kB/s | 116 kB 00:00 2023-11-15T05:50:00.097 INFO:teuthology.orchestra.run.smithi137.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 282 kB/s | 181 kB 00:00 2023-11-15T05:50:00.122 INFO:teuthology.orchestra.run.smithi137.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 387 kB/s | 306 kB 00:00 2023-11-15T05:50:00.139 INFO:teuthology.orchestra.run.smithi137.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 390 kB/s | 59 kB 00:00 2023-11-15T05:50:00.231 INFO:teuthology.orchestra.run.smithi137.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2023-11-15T05:50:00.257 INFO:teuthology.orchestra.run.smithi137.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 472 kB/s | 75 kB 00:00 2023-11-15T05:50:00.314 INFO:teuthology.orchestra.run.smithi137.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2023-11-15T05:50:00.319 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:00.320 INFO:teuthology.orchestra.run.smithi137.stdout:Total 20 MB/s | 59 MB 00:02 2023-11-15T05:50:00.387 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:50:00.435 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:50:00.436 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:50:00.473 INFO:teuthology.orchestra.run.smithi003.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2023-11-15T05:50:00.507 INFO:teuthology.orchestra.run.smithi003.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 744 kB/s | 25 kB 00:00 2023-11-15T05:50:00.580 INFO:teuthology.orchestra.run.smithi003.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 11 MB/s | 801 kB 00:00 2023-11-15T05:50:00.624 INFO:teuthology.orchestra.run.smithi003.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T05:50:00.733 INFO:teuthology.orchestra.run.smithi003.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-15T05:50:00.758 INFO:teuthology.orchestra.run.smithi003.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-15T05:50:00.769 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:50:00.772 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:50:00.817 INFO:teuthology.orchestra.run.smithi003.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.3 MB/s | 80 kB 00:00 2023-11-15T05:50:00.842 INFO:teuthology.orchestra.run.smithi003.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.5 MB/s | 213 kB 00:00 2023-11-15T05:50:00.876 INFO:teuthology.orchestra.run.smithi003.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T05:50:00.909 INFO:teuthology.orchestra.run.smithi003.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T05:50:00.934 INFO:teuthology.orchestra.run.smithi003.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T05:50:01.143 INFO:teuthology.orchestra.run.smithi003.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 15 MB/s | 13 MB 00:00 2023-11-15T05:50:01.218 INFO:teuthology.orchestra.run.smithi003.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 512 kB/s | 145 kB 00:00 2023-11-15T05:50:01.461 INFO:teuthology.orchestra.run.smithi003.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2023-11-15T05:50:01.485 INFO:teuthology.orchestra.run.smithi003.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-11-15T05:50:01.536 INFO:teuthology.orchestra.run.smithi003.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 817 kB/s | 259 kB 00:00 2023-11-15T05:50:01.569 INFO:teuthology.orchestra.run.smithi003.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2023-11-15T05:50:01.837 INFO:teuthology.orchestra.run.smithi003.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 120 kB/s | 42 kB 00:00 2023-11-15T05:50:01.874 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:50:01.887 INFO:teuthology.orchestra.run.smithi003.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 134 kB/s | 47 kB 00:00 2023-11-15T05:50:01.970 INFO:teuthology.orchestra.run.smithi003.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 200 kB/s | 80 kB 00:00 2023-11-15T05:50:02.054 INFO:teuthology.orchestra.run.smithi003.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 536 kB/s | 116 kB 00:00 2023-11-15T05:50:02.121 INFO:teuthology.orchestra.run.smithi003.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-11-15T05:50:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T05:50:02.146 INFO:teuthology.orchestra.run.smithi003.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 640 kB/s | 59 kB 00:00 2023-11-15T05:50:02.263 INFO:teuthology.orchestra.run.smithi003.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-11-15T05:50:02.288 INFO:teuthology.orchestra.run.smithi003.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 570 kB/s | 181 kB 00:00 2023-11-15T05:50:02.313 INFO:teuthology.orchestra.run.smithi003.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2023-11-15T05:50:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T05:50:02.389 INFO:teuthology.orchestra.run.smithi003.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.0 MB 00:00 2023-11-15T05:50:02.390 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:02.391 INFO:teuthology.orchestra.run.smithi003.stdout:Total 23 MB/s | 59 MB 00:02 2023-11-15T05:50:02.464 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:50:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T05:50:02.519 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:50:02.519 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:50:02.898 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:50:02.900 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T05:50:02.902 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:50:03.077 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:03.182 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:03.416 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T05:50:03.552 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:03.602 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:03.818 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T05:50:03.968 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T05:50:04.145 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:50:04.375 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T05:50:04.389 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T05:50:04.560 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:04.573 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T05:50:04.634 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:04.721 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T05:50:04.807 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T05:50:04.993 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T05:50:05.201 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T05:50:05.267 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T05:50:05.404 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:05.424 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T05:50:05.513 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:05.578 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T05:50:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T05:50:05.772 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T05:50:05.836 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T05:50:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:06.018 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T05:50:06.028 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T05:50:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T05:50:06.595 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T05:50:06.627 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T05:50:06.714 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T05:50:06.897 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T05:50:07.056 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T05:50:07.095 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T05:50:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T05:50:07.312 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T05:50:07.510 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T05:50:07.552 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T05:50:07.693 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T05:50:07.712 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T05:50:07.784 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T05:50:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T05:50:07.979 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T05:50:08.093 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T05:50:08.197 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T05:50:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T05:50:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T05:50:08.653 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T05:50:09.125 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T05:50:09.287 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T05:50:09.417 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T05:50:09.626 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T05:50:09.886 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T05:50:10.078 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T05:50:10.154 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T05:50:10.376 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T05:50:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T05:50:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T05:50:10.910 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T05:50:11.605 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:11.904 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T05:50:12.005 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T05:50:12.159 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:13.905 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:14.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:14.944 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T05:50:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T05:50:15.373 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T05:50:15.536 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:30.428 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:30.428 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2023-11-15T05:50:30.428 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2023-11-15T05:50:30.428 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2023-11-15T05:50:30.429 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /run 2023-11-15T05:50:30.429 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2023-11-15T05:50:30.429 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2023-11-15T05:50:30.429 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2023-11-15T05:50:30.429 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2023-11-15T05:50:30.430 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2023-11-15T05:50:30.430 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:31.776 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T05:50:31.850 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T05:50:31.850 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T05:50:32.050 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T05:50:32.050 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:32.737 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:32.738 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T05:50:32.738 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T05:50:32.738 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T05:50:32.738 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:32.738 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T05:50:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T05:50:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T05:50:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:32.739 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T05:50:32.740 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T05:50:32.740 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T05:50:32.740 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T05:50:32.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T05:50:32.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T05:50:32.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T05:50:32.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T05:50:32.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T05:50:32.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T05:50:32.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T05:50:32.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T05:50:32.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T05:50:32.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T05:50:32.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T05:50:32.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T05:50:32.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:32.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T05:50:32.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T05:50:33.940 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:33.940 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:33.940 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2023-11-15T05:50:33.941 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.941 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:50:33.941 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.941 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.941 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.942 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.942 INFO:teuthology.orchestra.run.smithi137.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T05:50:33.942 INFO:teuthology.orchestra.run.smithi137.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T05:50:33.942 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T05:50:33.942 INFO:teuthology.orchestra.run.smithi137.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.943 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T05:50:33.944 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.944 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T05:50:33.944 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T05:50:33.944 INFO:teuthology.orchestra.run.smithi137.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T05:50:33.944 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.945 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:33.946 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:33.946 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:50:34.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2023-11-15T05:50:35.051 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:43 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:50:36.740 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:50:36.741 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:36.741 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:50:36.741 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:36.741 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:50:36.742 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T05:50:36.742 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:50:36.742 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T05:50:36.742 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T05:50:36.742 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T05:50:36.743 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T05:50:36.743 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T05:50:36.743 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:36.743 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:50:36.743 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:36.744 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T05:50:36.744 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:36.745 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 53 M 2023-11-15T05:50:36.745 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 229 M 2023-11-15T05:50:36.745 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:50:36.890 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.8 MB/s | 203 kB 00:00 2023-11-15T05:50:36.895 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:36.895 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-11-15T05:50:36.895 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-11-15T05:50:36.895 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-11-15T05:50:36.896 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2023-11-15T05:50:36.896 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-11-15T05:50:36.896 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-11-15T05:50:36.896 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-11-15T05:50:36.896 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-11-15T05:50:36.897 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-11-15T05:50:36.897 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:36.915 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.2 MB/s | 169 kB 00:00 2023-11-15T05:50:36.940 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-11-15T05:50:36.982 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2023-11-15T05:50:37.033 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-11-15T05:50:37.584 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2023-11-15T05:50:37.586 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:37.587 INFO:teuthology.orchestra.run.smithi137.stdout:Total 63 MB/s | 53 MB 00:00 2023-11-15T05:50:37.630 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:50:37.652 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:50:37.652 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:50:37.958 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:50:37.961 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:50:38.386 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T05:50:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:50:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T05:50:38.502 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T05:50:38.641 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T05:50:38.819 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T05:50:38.819 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:38.848 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T05:50:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T05:50:39.071 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T05:50:39.270 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T05:50:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T05:50:39.569 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:39.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T05:50:39.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T05:50:39.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T05:50:39.571 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T05:50:39.572 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T05:50:39.572 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T05:50:39.572 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T05:50:39.572 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T05:50:39.573 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T05:50:39.573 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T05:50:39.573 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T05:50:39.573 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T05:50:39.574 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T05:50:39.574 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T05:50:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T05:50:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T05:50:39.576 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T05:50:39.576 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T05:50:39.576 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T05:50:39.577 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T05:50:39.578 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T05:50:39.578 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T05:50:39.579 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T05:50:39.579 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T05:50:39.579 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T05:50:39.580 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T05:50:39.580 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T05:50:39.581 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T05:50:40.984 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T05:50:40.984 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:40.985 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-11-15T05:50:40.985 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.985 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:50:40.985 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.985 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T05:50:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T05:50:40.986 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T05:50:40.987 INFO:teuthology.orchestra.run.smithi003.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T05:50:40.987 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.987 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T05:50:40.987 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T05:50:40.987 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T05:50:40.988 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.988 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T05:50:40.988 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.988 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T05:50:40.988 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T05:50:40.989 INFO:teuthology.orchestra.run.smithi003.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T05:50:40.989 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.989 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.990 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.990 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T05:50:40.990 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.990 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.990 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T05:50:40.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:40.991 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:50:41.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2023-11-15T05:50:42.324 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:48 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:50:44.168 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:50:44.170 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:44.170 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:50:44.170 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:44.170 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:50:44.171 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T05:50:44.171 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:50:44.171 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T05:50:44.171 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T05:50:44.171 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:44.172 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-15T05:50:44.173 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:44.173 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 53 M 2023-11-15T05:50:44.174 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 229 M 2023-11-15T05:50:44.174 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:50:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T05:50:44.450 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 169 kB 00:00 2023-11-15T05:50:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.5 MB/s | 203 kB 00:00 2023-11-15T05:50:44.510 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T05:50:44.548 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-11-15T05:50:44.580 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-11-15T05:50:44.605 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-11-15T05:50:44.788 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T05:50:44.789 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T05:50:44.789 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T05:50:44.789 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T05:50:44.789 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T05:50:44.789 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T05:50:45.480 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 46 MB/s | 52 MB 00:01 2023-11-15T05:50:45.482 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:45.483 INFO:teuthology.orchestra.run.smithi003.stdout:Total 40 MB/s | 53 MB 00:01 2023-11-15T05:50:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T05:50:45.493 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:45.493 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:50:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T05:50:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T05:50:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T05:50:45.494 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:45.495 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:50:45.528 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:50:45.551 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:50:45.552 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:50:45.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2023-11-15T05:50:45.910 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:50:45.914 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:50:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:50:46.574 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:50:46.707 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T05:50:46.911 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T05:50:47.081 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T05:50:47.171 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T05:50:47.373 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T05:50:47.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T05:50:48.220 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:50:48.223 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:50:48.224 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:48.224 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:50:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T05:50:48.224 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:50:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T05:50:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T05:50:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T05:50:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T05:50:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T05:50:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T05:50:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T05:50:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T05:50:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T05:50:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T05:50:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T05:50:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T05:50:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T05:50:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T05:50:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T05:50:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T05:50:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T05:50:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T05:50:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T05:50:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T05:50:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T05:50:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:48.230 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:50:48.231 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:50:48.231 INFO:teuthology.orchestra.run.smithi137.stdout:Install 32 Packages 2023-11-15T05:50:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:50:48.232 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 M 2023-11-15T05:50:48.232 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 111 M 2023-11-15T05:50:48.233 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:50:48.594 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 159 kB/s | 6.3 kB 00:00 2023-11-15T05:50:48.634 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2023-11-15T05:50:48.685 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2023-11-15T05:50:48.718 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-15T05:50:48.786 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2023-11-15T05:50:48.852 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 584 kB/s | 39 kB 00:00 2023-11-15T05:50:49.044 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2023-11-15T05:50:49.091 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 421 kB/s | 157 kB 00:00 2023-11-15T05:50:49.120 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 387 kB/s | 103 kB 00:00 2023-11-15T05:50:49.154 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 7.2 MB/s | 457 kB 00:00 2023-11-15T05:50:49.178 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-pytz-2017.2-11.el8.noarch.rpm 425 kB/s | 57 kB 00:00 2023-11-15T05:50:49.620 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 247 kB/s | 109 kB 00:00 2023-11-15T05:50:49.646 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 473 kB/s | 249 kB 00:00 2023-11-15T05:50:49.821 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 218 kB/s | 44 kB 00:00 2023-11-15T05:50:49.888 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 873 kB/s | 641 kB 00:00 2023-11-15T05:50:49.914 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 693 kB/s | 185 kB 00:00 2023-11-15T05:50:49.955 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-11-15T05:50:49.981 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2023-11-15T05:50:50.023 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-11-15T05:50:50.048 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 728 kB/s | 49 kB 00:00 2023-11-15T05:50:50.090 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 881 kB/s | 59 kB 00:00 2023-11-15T05:50:50.148 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-11-15T05:50:50.174 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-11-15T05:50:50.199 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2023-11-15T05:50:50.224 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 247 kB/s | 19 kB 00:00 2023-11-15T05:50:50.250 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-11-15T05:50:50.275 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2023-11-15T05:50:50.300 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-11-15T05:50:50.325 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T05:50:50.351 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-11-15T05:50:50.376 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-11-15T05:50:50.401 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-11-15T05:50:50.402 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:50.403 INFO:teuthology.orchestra.run.smithi137.stdout:Total 14 MB/s | 31 MB 00:02 2023-11-15T05:50:50.487 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:50:50.527 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:50:50.527 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:50:50.851 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:50:50.854 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:50:51.960 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:50:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T05:50:52.360 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T05:50:52.592 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T05:50:52.800 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T05:50:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T05:50:52.990 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T05:50:53.162 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T05:50:53.229 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T05:50:53.331 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T05:50:53.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T05:50:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T05:50:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T05:50:53.557 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T05:50:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T05:50:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T05:50:53.558 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T05:50:53.659 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T05:50:53.904 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T05:50:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T05:50:54.374 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T05:50:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T05:50:54.409 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:54.409 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:50:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T05:50:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T05:50:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T05:50:54.410 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:54.410 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:50:54.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2023-11-15T05:50:54.765 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T05:50:55.022 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T05:50:55.201 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T05:50:55.476 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T05:50:55.636 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T05:50:55.682 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:01 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:50:55.910 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T05:50:56.132 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T05:50:56.304 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T05:50:56.505 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T05:50:56.745 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T05:50:56.989 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T05:50:57.202 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T05:50:57.528 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T05:50:57.558 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:50:57.561 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:50:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:50:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T05:50:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:50:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T05:50:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T05:50:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T05:50:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T05:50:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T05:50:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T05:50:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T05:50:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T05:50:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T05:50:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T05:50:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T05:50:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T05:50:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T05:50:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T05:50:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T05:50:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T05:50:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T05:50:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T05:50:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T05:50:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T05:50:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T05:50:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T05:50:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T05:50:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T05:50:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T05:50:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T05:50:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:50:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:50:57.569 INFO:teuthology.orchestra.run.smithi003.stdout:Install 32 Packages 2023-11-15T05:50:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:50:57.570 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 M 2023-11-15T05:50:57.571 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 111 M 2023-11-15T05:50:57.571 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:50:57.764 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 110 kB/s | 6.3 kB 00:00 2023-11-15T05:50:57.823 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T05:50:57.844 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-11-15T05:50:57.907 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2023-11-15T05:50:57.932 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2023-11-15T05:50:58.181 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 12 MB/s | 5.0 MB 00:00 2023-11-15T05:50:58.184 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T05:50:58.217 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 554 kB/s | 157 kB 00:00 2023-11-15T05:50:58.258 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2023-11-15T05:50:58.283 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 382 kB/s | 39 kB 00:00 2023-11-15T05:50:58.308 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-11-15T05:50:58.405 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T05:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.8 MB/s | 457 kB 00:00 2023-11-15T05:50:58.618 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 24 MB/s | 18 MB 00:00 2023-11-15T05:50:58.901 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 420 kB/s | 249 kB 00:00 2023-11-15T05:50:59.027 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 349 kB/s | 44 kB 00:00 2023-11-15T05:50:59.052 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 251 kB/s | 109 kB 00:00 2023-11-15T05:50:59.135 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.7 MB/s | 185 kB 00:00 2023-11-15T05:50:59.194 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): python3-cryptography-3.2.1-6.el8.x86_6 854 kB/s | 641 kB 00:00 2023-11-15T05:50:59.219 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2023-11-15T05:50:59.252 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 861 kB/s | 173 kB 00:00 2023-11-15T05:50:59.278 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 329 kB/s | 18 kB 00:00 2023-11-15T05:50:59.303 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-11-15T05:50:59.357 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 471 kB/s | 49 kB 00:00 2023-11-15T05:50:59.387 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 541 kB/s | 59 kB 00:00 2023-11-15T05:50:59.446 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-11-15T05:50:59.471 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 137 kB/s | 16 kB 00:00 2023-11-15T05:50:59.496 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2023-11-15T05:50:59.521 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2023-11-15T05:50:59.547 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2023-11-15T05:50:59.594 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 284 kB/s | 27 kB 00:00 2023-11-15T05:50:59.630 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2023-11-15T05:50:59.656 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2023-11-15T05:50:59.681 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-11-15T05:50:59.706 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 761 kB/s | 85 kB 00:00 2023-11-15T05:50:59.710 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:50:59.711 INFO:teuthology.orchestra.run.smithi003.stdout:Total 14 MB/s | 31 MB 00:02 2023-11-15T05:50:59.801 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:50:59.844 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:50:59.844 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:50:59.940 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T05:51:00.104 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T05:51:00.169 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:00.172 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:00.539 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T05:51:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T05:51:01.003 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T05:51:01.168 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T05:51:01.186 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T05:51:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T05:51:01.429 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:01.661 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T05:51:01.741 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T05:51:01.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T05:51:01.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T05:51:01.741 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T05:51:01.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T05:51:01.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T05:51:01.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T05:51:01.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T05:51:01.742 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T05:51:01.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T05:51:01.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T05:51:01.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T05:51:01.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T05:51:01.743 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T05:51:01.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T05:51:01.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T05:51:01.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T05:51:01.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T05:51:01.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T05:51:01.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T05:51:01.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T05:51:01.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T05:51:01.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T05:51:01.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T05:51:01.747 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T05:51:01.747 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T05:51:01.747 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T05:51:01.889 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T05:51:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T05:51:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T05:51:02.584 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T05:51:02.792 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T05:51:02.849 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T05:51:02.849 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:02.849 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:02.849 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:02.850 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:02.850 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:02.850 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T05:51:02.850 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:02.851 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:02.851 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T05:51:02.851 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T05:51:02.851 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T05:51:02.851 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T05:51:02.852 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T05:51:02.852 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T05:51:02.852 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T05:51:02.852 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T05:51:02.852 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T05:51:02.853 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T05:51:02.853 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T05:51:02.853 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T05:51:02.853 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T05:51:02.853 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T05:51:02.854 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T05:51:02.854 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T05:51:02.854 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T05:51:02.854 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T05:51:02.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T05:51:02.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T05:51:02.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T05:51:02.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T05:51:02.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T05:51:02.856 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T05:51:02.856 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T05:51:02.856 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T05:51:02.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:02.857 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:02.982 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T05:51:03.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2023-11-15T05:51:03.168 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T05:51:03.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T05:51:03.633 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T05:51:03.923 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T05:51:03.985 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:11 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:04.205 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T05:51:04.588 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T05:51:04.894 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T05:51:05.113 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T05:51:05.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T05:51:05.419 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:51:05.606 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T05:51:05.638 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:05.639 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:51:05.639 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:05.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2023-11-15T05:51:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T05:51:06.137 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T05:51:06.341 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T05:51:06.623 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:06.736 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T05:51:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T05:51:07.297 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T05:51:07.567 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T05:51:07.938 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T05:51:08.265 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:08.266 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:51:08.267 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:08.267 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T05:51:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:51:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 71 k 2023-11-15T05:51:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 290 k 2023-11-15T05:51:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:08.276 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T05:51:08.314 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-11-15T05:51:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-11-15T05:51:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:08.322 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:08.322 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:08.323 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:08.326 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:08.685 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T05:51:08.863 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:08.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T05:51:09.074 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:09.306 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:09.948 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:09.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:09.949 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:09.949 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T05:51:09.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:09.950 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:10.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T05:51:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T05:51:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T05:51:11.077 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:19 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T05:51:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T05:51:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T05:51:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T05:51:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T05:51:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T05:51:12.745 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:12.746 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:12.746 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2023-11-15T05:51:12.746 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:12.746 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:12.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T05:51:12.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:12.747 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:12.747 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:12.747 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:51:12.748 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:12.748 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 163 k 2023-11-15T05:51:12.749 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 587 k 2023-11-15T05:51:12.749 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:12.794 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-11-15T05:51:12.794 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:12.795 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-11-15T05:51:12.795 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:12.805 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:12.805 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:12.852 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:12.855 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:12.944 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T05:51:12.944 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T05:51:12.945 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T05:51:12.945 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T05:51:12.945 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T05:51:12.945 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T05:51:12.945 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T05:51:12.946 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T05:51:12.946 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T05:51:12.946 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T05:51:12.946 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T05:51:12.946 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T05:51:12.947 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T05:51:12.947 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T05:51:12.947 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T05:51:12.947 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T05:51:12.947 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T05:51:12.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T05:51:12.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T05:51:12.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T05:51:12.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T05:51:12.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T05:51:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T05:51:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T05:51:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T05:51:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T05:51:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T05:51:12.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T05:51:12.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T05:51:12.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T05:51:12.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T05:51:12.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T05:51:13.000 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:13.149 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:13.555 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:14.133 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T05:51:14.133 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:14.134 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:51:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.134 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T05:51:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T05:51:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T05:51:14.135 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T05:51:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T05:51:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T05:51:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T05:51:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T05:51:14.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T05:51:14.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T05:51:14.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T05:51:14.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T05:51:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T05:51:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T05:51:14.139 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T05:51:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T05:51:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T05:51:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T05:51:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T05:51:14.140 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T05:51:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T05:51:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T05:51:14.141 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T05:51:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T05:51:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T05:51:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T05:51:14.142 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T05:51:14.143 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:14.143 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:14.237 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:14.238 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:14.238 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:14.238 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:14.238 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:14.239 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:14.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2023-11-15T05:51:14.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2023-11-15T05:51:15.383 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:23 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:15.437 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:21 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:16.824 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:51:17.044 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:17.045 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:51:17.045 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:17.057 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:51:17.301 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:17.302 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:51:17.303 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:17.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T05:51:17.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2023-11-15T05:51:18.255 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:18.396 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:19.933 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:19.934 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:51:19.934 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:19.934 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T05:51:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T05:51:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T05:51:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:19.936 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:19.936 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:19.936 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T05:51:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:19.937 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.0 M 2023-11-15T05:51:19.938 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 15 M 2023-11-15T05:51:19.938 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:20.088 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 186 kB/s | 8.9 kB 00:00 2023-11-15T05:51:20.105 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 367 kB/s | 24 kB 00:00 2023-11-15T05:51:20.158 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 23 MB/s | 2.7 MB 00:00 2023-11-15T05:51:20.236 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:20.237 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:20.237 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:51:20.237 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:20.237 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:51:20.238 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T05:51:20.238 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:20.238 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:51:20.239 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:20.239 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:51:20.239 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:20.239 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 71 k 2023-11-15T05:51:20.239 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 290 k 2023-11-15T05:51:20.240 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:51:20.287 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-11-15T05:51:20.288 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:20.288 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.4 MB/s | 71 kB 00:00 2023-11-15T05:51:20.288 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:51:20.296 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:51:20.296 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:51:20.298 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:20.302 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:20.369 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:20.456 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 116 kB/s | 43 kB 00:00 2023-11-15T05:51:20.481 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 90 kB/s | 34 kB 00:00 2023-11-15T05:51:20.714 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 351 kB/s | 196 kB 00:00 2023-11-15T05:51:20.717 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:20.717 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.8 MB/s | 3.0 MB 00:00 2023-11-15T05:51:20.728 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:20.737 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:20.737 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:20.898 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:20.902 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:20.916 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:21.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:21.389 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:21.395 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T05:51:21.640 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T05:51:21.824 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T05:51:21.965 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T05:51:22.111 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T05:51:22.111 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:22.112 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:51:22.112 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T05:51:22.112 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:22.112 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:22.951 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T05:51:23.208 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:23.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T05:51:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T05:51:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T05:51:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T05:51:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T05:51:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T05:51:24.166 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T05:51:24.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:24.166 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:24.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T05:51:24.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T05:51:24.167 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T05:51:24.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T05:51:24.167 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T05:51:24.168 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T05:51:24.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:24.168 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:24.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T05:51:24.511 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:30 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:25.167 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:33 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:26.380 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:26.382 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:26.382 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-11-15T05:51:26.382 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:26.382 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:51:26.383 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T05:51:26.383 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:26.383 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:51:26.383 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:26.383 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:51:26.384 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:26.384 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 163 k 2023-11-15T05:51:26.384 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 587 k 2023-11-15T05:51:26.384 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:51:26.434 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.1 MB/s | 163 kB 00:00 2023-11-15T05:51:26.435 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:26.436 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.0 MB/s | 163 kB 00:00 2023-11-15T05:51:26.436 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:51:26.448 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:51:26.449 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:51:26.503 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:26.507 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:26.641 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:26.850 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:26.851 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:26.853 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T05:51:26.853 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:51:26.854 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T05:51:26.854 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:26.854 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T05:51:26.854 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:51:26.854 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T05:51:26.855 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T05:51:26.855 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T05:51:26.855 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T05:51:26.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T05:51:26.855 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================================ 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout:Install 8 Packages 2023-11-15T05:51:26.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:26.857 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 37 M 2023-11-15T05:51:26.857 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 210 M 2023-11-15T05:51:26.858 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:27.086 INFO:teuthology.orchestra.run.smithi137.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 56 MB/s | 8.6 MB 00:00 2023-11-15T05:51:27.278 INFO:teuthology.orchestra.run.smithi137.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2023-11-15T05:51:27.323 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:27.345 INFO:teuthology.orchestra.run.smithi137.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2023-11-15T05:51:27.370 INFO:teuthology.orchestra.run.smithi137.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-15T05:51:27.429 INFO:teuthology.orchestra.run.smithi137.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2023-11-15T05:51:27.721 INFO:teuthology.orchestra.run.smithi137.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 36 MB/s | 14 MB 00:00 2023-11-15T05:51:27.946 INFO:teuthology.orchestra.run.smithi137.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 334 kB/s | 172 kB 00:00 2023-11-15T05:51:28.061 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:51:28.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:28.062 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:51:28.062 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T05:51:28.062 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:28.062 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:28.142 INFO:teuthology.orchestra.run.smithi137.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 836 kB/s | 645 kB 00:00 2023-11-15T05:51:28.143 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:28.143 INFO:teuthology.orchestra.run.smithi137.stdout:Total 28 MB/s | 37 MB 00:01 2023-11-15T05:51:28.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2023-11-15T05:51:28.376 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:28.395 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:28.395 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:28.715 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:28.719 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:29.334 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:35 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:29.471 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T05:51:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T05:51:29.754 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T05:51:30.355 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T05:51:30.562 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:30.943 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:51:31.164 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:31.185 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:31.186 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:51:31.187 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:31.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T05:51:31.337 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T05:51:32.020 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T05:51:32.295 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:32.348 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T05:51:32.574 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T05:51:34.142 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:34.144 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:51:34.144 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:34.144 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:51:34.144 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T05:51:34.145 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:51:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T05:51:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T05:51:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T05:51:34.145 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T05:51:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T05:51:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:34.146 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:51:34.146 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:34.147 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-15T05:51:34.147 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:34.148 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.0 M 2023-11-15T05:51:34.148 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 15 M 2023-11-15T05:51:34.149 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:51:34.327 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T05:51:34.795 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 443 kB/s | 24 kB 00:00 2023-11-15T05:51:34.811 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 126 kB/s | 8.9 kB 00:00 2023-11-15T05:51:34.870 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-11-15T05:51:34.954 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-11-15T05:51:35.037 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-11-15T05:51:35.171 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 113 kB/s | 43 kB 00:00 2023-11-15T05:51:35.173 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:35.174 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.9 MB/s | 3.0 MB 00:01 2023-11-15T05:51:35.187 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:51:35.199 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:51:35.200 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:51:35.398 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:35.402 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:35.644 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T05:51:35.786 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T05:51:36.036 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T05:51:36.174 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T05:51:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T05:51:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T05:51:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T05:51:36.175 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T05:51:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T05:51:36.176 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T05:51:36.287 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T05:51:36.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T05:51:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T05:51:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T05:51:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:36.873 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T05:51:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T05:51:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T05:51:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T05:51:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T05:51:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T05:51:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T05:51:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T05:51:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:36.875 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:36.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2023-11-15T05:51:37.778 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T05:51:37.862 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:45 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:38.043 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:38.517 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:38.517 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T05:51:38.517 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T05:51:38.518 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T05:51:38.518 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T05:51:38.518 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T05:51:39.429 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T05:51:39.429 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:39.429 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:51:39.430 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T05:51:39.431 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T05:51:39.431 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T05:51:39.431 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T05:51:39.432 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T05:51:39.432 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T05:51:39.432 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:39.506 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:39.507 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:39.507 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:51:39.508 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:39.508 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:39.508 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T05:51:39.508 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:51:39.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T05:51:39.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T05:51:39.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T05:51:39.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T05:51:39.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T05:51:39.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T05:51:39.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T05:51:39.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T05:51:39.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T05:51:39.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T05:51:39.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T05:51:39.511 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T05:51:39.514 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:39.515 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:39.515 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:39.515 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2023-11-15T05:51:39.515 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:39.516 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 1.9 M 2023-11-15T05:51:39.516 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 24 M 2023-11-15T05:51:39.517 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:39.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T05:51:40.123 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 653 kB/s | 27 kB 00:00 2023-11-15T05:51:40.175 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 219 kB/s | 20 kB 00:00 2023-11-15T05:51:40.197 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 228 kB/s | 27 kB 00:00 2023-11-15T05:51:40.231 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.8 MB/s | 110 kB 00:00 2023-11-15T05:51:40.256 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 943 kB/s | 126 kB 00:00 2023-11-15T05:51:40.392 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 247 kB/s | 33 kB 00:00 2023-11-15T05:51:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2023-11-15T05:51:40.571 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:40.574 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 124 kB/s | 43 kB 00:00 2023-11-15T05:51:40.600 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 572 kB/s | 81 kB 00:00 2023-11-15T05:51:40.692 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 314 kB/s | 155 kB 00:00 2023-11-15T05:51:40.759 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 404 kB/s | 64 kB 00:00 2023-11-15T05:51:40.793 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 5.4 MB/s | 1.2 MB 00:00 2023-11-15T05:51:40.854 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 386 kB/s | 61 kB 00:00 2023-11-15T05:51:40.855 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:40.855 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-11-15T05:51:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:40.926 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:40.927 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:41.013 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:41.016 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:41.525 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:41.747 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T05:51:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T05:51:42.163 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T05:51:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T05:51:42.423 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:42.424 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-15T05:51:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:51:42.424 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-15T05:51:42.425 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:51:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T05:51:42.425 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:51:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T05:51:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T05:51:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T05:51:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T05:51:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T05:51:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T05:51:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T05:51:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:42.427 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:51:42.427 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-15T05:51:42.427 INFO:teuthology.orchestra.run.smithi003.stdout:Install 8 Packages 2023-11-15T05:51:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:42.428 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2023-11-15T05:51:42.429 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 210 M 2023-11-15T05:51:42.429 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:51:42.587 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T05:51:42.693 INFO:teuthology.orchestra.run.smithi003.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 45 MB/s | 8.6 MB 00:00 2023-11-15T05:51:42.811 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T05:51:42.870 INFO:teuthology.orchestra.run.smithi003.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-11-15T05:51:42.979 INFO:teuthology.orchestra.run.smithi003.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2023-11-15T05:51:43.004 INFO:teuthology.orchestra.run.smithi003.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-11-15T05:51:43.142 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T05:51:43.330 INFO:teuthology.orchestra.run.smithi003.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 39 MB/s | 14 MB 00:00 2023-11-15T05:51:43.400 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T05:51:43.422 INFO:teuthology.orchestra.run.smithi003.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2023-11-15T05:51:43.688 INFO:teuthology.orchestra.run.smithi003.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 943 kB/s | 645 kB 00:00 2023-11-15T05:51:43.838 INFO:teuthology.orchestra.run.smithi003.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 339 kB/s | 172 kB 00:00 2023-11-15T05:51:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:Total 26 MB/s | 37 MB 00:01 2023-11-15T05:51:44.088 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:51:44.108 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:51:44.109 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:51:44.412 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T05:51:44.473 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:44.477 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:44.684 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T05:51:44.871 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T05:51:45.042 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T05:51:45.154 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:45.235 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T05:51:45.338 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T05:51:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T05:51:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T05:51:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T05:51:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T05:51:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T05:51:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T05:51:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T05:51:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T05:51:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T05:51:45.713 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T05:51:46.259 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T05:51:46.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:46.259 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:46.260 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T05:51:46.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T05:51:46.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T05:51:46.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T05:51:46.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T05:51:46.261 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T05:51:46.261 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T05:51:46.261 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T05:51:46.261 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T05:51:46.261 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T05:51:46.262 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T05:51:46.262 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T05:51:46.262 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T05:51:46.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:46.262 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:46.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T05:51:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T05:51:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:47.254 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:47.310 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:47.621 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T05:51:48.382 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T05:51:48.878 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T05:51:48.920 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:48.920 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:48.920 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2023-11-15T05:51:48.920 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:48.921 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:48.921 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T05:51:48.921 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:51:48.921 INFO:teuthology.orchestra.run.smithi137.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T05:51:48.922 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T05:51:48.922 INFO:teuthology.orchestra.run.smithi137.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T05:51:48.922 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T05:51:48.922 INFO:teuthology.orchestra.run.smithi137.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T05:51:48.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:48.923 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:48.923 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:48.923 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2023-11-15T05:51:48.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:48.924 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 5.8 M 2023-11-15T05:51:48.924 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 24 M 2023-11-15T05:51:48.924 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:49.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T05:51:49.351 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-11-15T05:51:49.493 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.8 MB/s | 538 kB 00:00 2023-11-15T05:51:49.785 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 49 kB/s | 21 kB 00:00 2023-11-15T05:51:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.3 MB/s | 4.8 MB 00:00 2023-11-15T05:51:49.980 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2023-11-15T05:51:50.335 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 422 kB/s | 355 kB 00:00 2023-11-15T05:51:50.337 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:50.338 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.1 MB/s | 5.8 MB 00:01 2023-11-15T05:51:50.392 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:50.402 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:50.402 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:50.507 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:50.510 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:50.797 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:51.042 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T05:51:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T05:51:51.339 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T05:51:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T05:51:52.608 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T05:51:52.639 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T05:51:52.825 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T05:51:52.828 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T05:51:53.034 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:53.197 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T05:51:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T05:51:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T05:51:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T05:51:53.198 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T05:51:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T05:51:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T05:51:53.199 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T05:51:53.324 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:51:53.324 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T05:51:53.324 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T05:51:53.324 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T05:51:53.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T05:51:53.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T05:51:54.006 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T05:51:54.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:54.007 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:54.007 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T05:51:54.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T05:51:54.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T05:51:54.008 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:54.008 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T05:51:54.024 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:54.025 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:51:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T05:51:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T05:51:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T05:51:54.025 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T05:51:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T05:51:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T05:51:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T05:51:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T05:51:54.026 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:54.027 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:51:54.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2023-11-15T05:51:54.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2023-11-15T05:51:55.023 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:51:55.157 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:01 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:51:56.702 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:51:56.703 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:56.703 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:51:56.703 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:56.703 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:51:56.704 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T05:51:56.704 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2023-11-15T05:51:56.704 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T05:51:56.704 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:56.704 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:51:56.705 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:51:56.705 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2023-11-15T05:51:56.705 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:56.705 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 968 k 2023-11-15T05:51:56.705 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 3.2 M 2023-11-15T05:51:56.706 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:51:56.947 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T05:51:57.010 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:51:57.012 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:57.012 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:51:57.013 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:57.013 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:51:57.013 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T05:51:57.013 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:51:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T05:51:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T05:51:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T05:51:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T05:51:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T05:51:57.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T05:51:57.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T05:51:57.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T05:51:57.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T05:51:57.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:51:57.016 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-11-15T05:51:57.017 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:51:57.017 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.9 M 2023-11-15T05:51:57.018 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 24 M 2023-11-15T05:51:57.018 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:51:57.313 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 198 kB/s | 83 kB 00:00 2023-11-15T05:51:57.314 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:57.314 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.5 MB/s | 968 kB 00:00 2023-11-15T05:51:57.325 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:51:57.345 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:51:57.346 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:51:57.430 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:51:57.434 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:51:57.494 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 529 kB/s | 27 kB 00:00 2023-11-15T05:51:57.552 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 183 kB/s | 20 kB 00:00 2023-11-15T05:51:57.578 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 197 kB/s | 27 kB 00:00 2023-11-15T05:51:57.603 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:51:57.611 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-11-15T05:51:57.637 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.3 MB/s | 110 kB 00:00 2023-11-15T05:51:57.787 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2023-11-15T05:51:57.829 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 391 kB/s | 16 kB 00:00 2023-11-15T05:51:57.841 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T05:51:57.896 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2023-11-15T05:51:57.946 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 127 kB/s | 43 kB 00:00 2023-11-15T05:51:58.000 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T05:51:58.030 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 8.7 MB/s | 1.2 MB 00:00 2023-11-15T05:51:58.055 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 325 kB/s | 155 kB 00:00 2023-11-15T05:51:58.080 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2023-11-15T05:51:58.114 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 384 kB/s | 64 kB 00:00 2023-11-15T05:51:58.116 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:51:58.116 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2023-11-15T05:51:58.179 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:51:58.197 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:51:58.197 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:51:58.297 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:51:58.300 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:51:58.383 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T05:51:58.384 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T05:51:58.886 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:51:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T05:51:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:59.056 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:51:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T05:51:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:51:59.057 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:51:59.156 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T05:51:59.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2023-11-15T05:51:59.368 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T05:51:59.669 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T05:51:59.928 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T05:52:00.145 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:00.193 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T05:52:00.456 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T05:52:00.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T05:52:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T05:52:01.792 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:01.793 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:01.793 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:01.793 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:01.793 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:52:01.794 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T05:52:01.794 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:01.794 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:52:01.794 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:01.794 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:52:01.795 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:01.795 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 131 k 2023-11-15T05:52:01.795 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 439 k 2023-11-15T05:52:01.795 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:52:01.841 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2023-11-15T05:52:01.842 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:01.842 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.6 MB/s | 131 kB 00:00 2023-11-15T05:52:01.843 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:52:01.848 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:52:01.848 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:52:01.897 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:52:01.900 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:52:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:52:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T05:52:02.570 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T05:52:02.936 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T05:52:03.170 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T05:52:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:03.253 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:52:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:03.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:03.254 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:03.369 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T05:52:03.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2023-11-15T05:52:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T05:52:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T05:52:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T05:52:03.713 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T05:52:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T05:52:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T05:52:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T05:52:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T05:52:03.714 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T05:52:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T05:52:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T05:52:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T05:52:03.715 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T05:52:04.335 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:12 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:04.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T05:52:04.732 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:04.732 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:04.732 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T05:52:04.733 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T05:52:04.733 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T05:52:04.733 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T05:52:04.733 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T05:52:04.733 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T05:52:04.734 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T05:52:04.734 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T05:52:04.734 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T05:52:04.734 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T05:52:04.735 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T05:52:04.735 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T05:52:04.735 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T05:52:04.735 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:04.735 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:04.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T05:52:05.790 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:05.880 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:11 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:06.005 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:06.006 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:06.007 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:06.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2023-11-15T05:52:07.352 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:07.736 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:07.736 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:52:07.737 INFO:teuthology.orchestra.run.smithi003.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T05:52:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T05:52:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T05:52:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T05:52:07.738 INFO:teuthology.orchestra.run.smithi003.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T05:52:07.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:07.739 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:52:07.739 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:07.739 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-15T05:52:07.740 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:07.740 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.8 M 2023-11-15T05:52:07.741 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 24 M 2023-11-15T05:52:07.741 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:52:09.025 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:09.026 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:09.026 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:09.026 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:09.026 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:52:09.027 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T05:52:09.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:09.027 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:52:09.027 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:09.027 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:52:09.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:09.028 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 24 k 2023-11-15T05:52:09.028 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 94 k 2023-11-15T05:52:09.028 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:52:09.104 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 315 kB/s | 24 kB 00:00 2023-11-15T05:52:09.104 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:09.105 INFO:teuthology.orchestra.run.smithi137.stdout:Total 310 kB/s | 24 kB 00:00 2023-11-15T05:52:09.105 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:52:09.109 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:52:09.109 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:52:09.110 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:52:09.113 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:52:09.169 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:52:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:10.272 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:10.272 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:10.272 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:52:10.273 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:10.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:10.273 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:10.321 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2023-11-15T05:52:10.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2023-11-15T05:52:10.438 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.4 MB/s | 538 kB 00:00 2023-11-15T05:52:10.638 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-11-15T05:52:11.189 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 24 kB/s | 21 kB 00:00 2023-11-15T05:52:11.366 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:11.705 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 46 kB/s | 49 kB 00:01 2023-11-15T05:52:12.130 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 210 kB/s | 355 kB 00:01 2023-11-15T05:52:12.134 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:12.134 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.3 MB/s | 5.8 MB 00:04 2023-11-15T05:52:12.191 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:52:12.202 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:52:12.203 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:52:12.321 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:52:12.325 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:52:12.657 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:52:12.722 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:12.944 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:12.945 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:12.945 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:12.988 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T05:52:13.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2023-11-15T05:52:13.310 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T05:52:13.928 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:21 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:14.349 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T05:52:14.731 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T05:52:14.993 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T05:52:15.333 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:52:15.376 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:15.591 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:15.592 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:15.593 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:15.649 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T05:52:15.650 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T05:52:15.650 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T05:52:15.650 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T05:52:15.651 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T05:52:15.651 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T05:52:15.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2023-11-15T05:52:16.457 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T05:52:16.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:16.457 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:16.457 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T05:52:16.458 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T05:52:16.458 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T05:52:16.458 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:16.458 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:16.568 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:16.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2023-11-15T05:52:17.605 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:18.030 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:18.246 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:18.247 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:18.247 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:18.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2023-11-15T05:52:19.227 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:19.468 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:19.468 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:19.469 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:19.469 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:52:19.469 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T05:52:19.469 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-15T05:52:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T05:52:19.470 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:19.470 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:52:19.470 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:19.470 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2023-11-15T05:52:19.471 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:19.471 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 968 k 2023-11-15T05:52:19.472 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 3.2 M 2023-11-15T05:52:19.472 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:52:19.756 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 9.1 MB/s | 885 kB 00:00 2023-11-15T05:52:20.099 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 189 kB/s | 83 kB 00:00 2023-11-15T05:52:20.100 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:20.100 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.5 MB/s | 968 kB 00:00 2023-11-15T05:52:20.113 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:52:20.138 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:52:20.138 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:52:20.240 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:52:20.244 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:52:20.433 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:52:20.661 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:20.717 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T05:52:20.875 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:20.876 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:20.876 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:20.951 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T05:52:20.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2023-11-15T05:52:21.411 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T05:52:21.411 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T05:52:21.852 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:22.185 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T05:52:22.185 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:22.185 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:22.186 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T05:52:22.186 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:22.186 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:22.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2023-11-15T05:52:23.305 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:23.478 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:23.523 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:23.524 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:23.525 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:23.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2023-11-15T05:52:24.525 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:25.327 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:25.327 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:25.328 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:25.328 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:25.328 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:52:25.328 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T05:52:25.328 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:25.329 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:52:25.329 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:25.329 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:52:25.329 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:25.330 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 131 k 2023-11-15T05:52:25.330 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 439 k 2023-11-15T05:52:25.330 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:52:25.390 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 131 kB 00:00 2023-11-15T05:52:25.390 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:25.391 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.1 MB/s | 131 kB 00:00 2023-11-15T05:52:25.391 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:52:25.398 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:52:25.398 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:52:25.454 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:52:25.458 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:52:25.578 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:52:25.816 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:25.981 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:26.135 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:26.199 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:26.200 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2023-11-15T05:52:26.200 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:26.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2023-11-15T05:52:26.902 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:26.902 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:26.902 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:26.902 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:26.903 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:26.903 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:27.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2023-11-15T05:52:27.204 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:28.136 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:34 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:28.857 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:28.858 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:28.858 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:28.858 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:28.859 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:52:28.859 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T05:52:28.859 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:28.859 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:52:28.859 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:28.860 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:52:28.860 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:28.860 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 89 k 2023-11-15T05:52:28.860 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 298 k 2023-11-15T05:52:28.860 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:52:28.931 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.2 MB/s | 89 kB 00:00 2023-11-15T05:52:28.932 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:28.932 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 89 kB 00:00 2023-11-15T05:52:28.932 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:52:28.944 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:52:28.944 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:52:28.994 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:52:29.003 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:52:29.287 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:52:29.471 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:29.754 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:29.754 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:29.996 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:29.997 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:30.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2023-11-15T05:52:30.424 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:30.424 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:30.424 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:52:30.425 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:30.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:30.425 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:30.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2023-11-15T05:52:31.107 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:37 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:31.508 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:32.968 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:32.968 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:32.969 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:32.969 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:32.969 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:52:32.969 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T05:52:32.969 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:32.970 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:52:32.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:32.970 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:52:32.970 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:32.971 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 24 k 2023-11-15T05:52:32.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 94 k 2023-11-15T05:52:32.971 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:52:33.017 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 514 kB/s | 24 kB 00:00 2023-11-15T05:52:33.018 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:33.018 INFO:teuthology.orchestra.run.smithi003.stdout:Total 500 kB/s | 24 kB 00:00 2023-11-15T05:52:33.019 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:52:33.022 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:52:33.023 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:52:33.025 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:52:33.028 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:52:33.078 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:52:33.173 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:33.173 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:33.174 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:33.174 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:33.174 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:52:33.174 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T05:52:33.174 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:33.175 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:52:33.175 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:33.175 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:52:33.175 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:33.176 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.7 M 2023-11-15T05:52:33.176 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 18 M 2023-11-15T05:52:33.176 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:52:33.265 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2023-11-15T05:52:33.265 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:33.266 INFO:teuthology.orchestra.run.smithi137.stdout:Total 41 MB/s | 3.7 MB 00:00 2023-11-15T05:52:33.266 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:52:33.278 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:52:33.278 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:52:33.340 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:52:33.343 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:52:33.456 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:33.743 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:33.802 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:52:33.978 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:34.405 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:34.499 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:34.499 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:34.500 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:34.500 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:34.500 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:34.500 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:34.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2023-11-15T05:52:35.116 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:35.117 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:35.117 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:52:35.117 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:35.117 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:35.118 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:35.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2023-11-15T05:52:35.627 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:41 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:36.182 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 05:49:52 AM UTC. 2023-11-15T05:52:37.235 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:37.476 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:37.478 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:37.478 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:37.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2023-11-15T05:52:37.833 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2023-11-15T05:52:37.834 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:37.834 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:37.834 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:37.834 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2023-11-15T05:52:37.834 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T05:52:37.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:37.835 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2023-11-15T05:52:37.835 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2023-11-15T05:52:37.835 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2023-11-15T05:52:37.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:37.836 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 173 k 2023-11-15T05:52:37.836 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 649 k 2023-11-15T05:52:37.836 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2023-11-15T05:52:37.875 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.1 MB/s | 173 kB 00:00 2023-11-15T05:52:37.876 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:37.876 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.0 MB/s | 173 kB 00:00 2023-11-15T05:52:37.877 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2023-11-15T05:52:37.887 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2023-11-15T05:52:37.887 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2023-11-15T05:52:37.937 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2023-11-15T05:52:37.941 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2023-11-15T05:52:38.052 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2023-11-15T05:52:38.217 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:38.596 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:39.143 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:39.144 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2023-11-15T05:52:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:52:39.144 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2023-11-15T05:52:39.350 DEBUG:teuthology.parallel:result is None 2023-11-15T05:52:40.215 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:40.455 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:40.457 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:40.457 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:40.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2023-11-15T05:52:41.569 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:47 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:43.179 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:43.420 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:43.421 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:43.421 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:43.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2023-11-15T05:52:44.535 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:50 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:46.146 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:46.387 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:46.389 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:46.389 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:46.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2023-11-15T05:52:47.495 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:53 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:49.100 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:49.340 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:49.341 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:49.341 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:49.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2023-11-15T05:52:50.436 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:56 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:52.044 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T05:52:52.287 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:52.289 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-15T05:52:52.289 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:52.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2023-11-15T05:52:53.419 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:59 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:52:55.270 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:52:55.275 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:55.275 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:52:55.275 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:55.276 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:52:55.276 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T05:52:55.277 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:55.277 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:52:55.277 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:52:55.277 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 89 k 2023-11-15T05:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 298 k 2023-11-15T05:52:55.278 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:52:55.349 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.1 MB/s | 89 kB 00:00 2023-11-15T05:52:55.350 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:52:55.350 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.1 MB/s | 89 kB 00:00 2023-11-15T05:52:55.350 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:52:55.362 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:52:55.362 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:52:55.419 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:52:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:52:55.550 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:52:55.886 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:56.214 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:56.989 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:52:56.989 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:56.989 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:52:56.990 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T05:52:56.990 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:52:56.990 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:52:57.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2023-11-15T05:52:58.181 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:04 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:53:00.035 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:53:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:00.036 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:53:00.036 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:53:00.037 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T05:53:00.037 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:53:00.037 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:00.038 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:53:00.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:00.038 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.7 M 2023-11-15T05:53:00.039 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 18 M 2023-11-15T05:53:00.039 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:53:00.125 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2023-11-15T05:53:00.126 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:53:00.126 INFO:teuthology.orchestra.run.smithi003.stdout:Total 42 MB/s | 3.7 MB 00:00 2023-11-15T05:53:00.127 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:53:00.140 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:53:00.141 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:53:00.214 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:53:00.214 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:53:00.756 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:53:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:01.690 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:02.481 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:53:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T05:53:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:02.482 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:53:02.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2023-11-15T05:53:03.678 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:09 ago on Wed 15 Nov 2023 05:49:54 AM UTC. 2023-11-15T05:53:05.549 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-15T05:53:05.550 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:05.550 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-15T05:53:05.550 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:05.550 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-15T05:53:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T05:53:05.551 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:05.551 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-15T05:53:05.551 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-15T05:53:05.551 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-15T05:53:05.552 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:05.552 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 173 k 2023-11-15T05:53:05.552 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 649 k 2023-11-15T05:53:05.553 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-15T05:53:05.604 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2023-11-15T05:53:05.604 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-15T05:53:05.605 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.2 MB/s | 173 kB 00:00 2023-11-15T05:53:05.605 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-15T05:53:05.617 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-15T05:53:05.617 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-15T05:53:05.674 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-15T05:53:05.677 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-15T05:53:05.801 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-15T05:53:06.026 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:06.359 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:07.107 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T05:53:07.107 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:07.108 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-15T05:53:07.108 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T05:53:07.108 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-15T05:53:07.108 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-15T05:53:07.359 DEBUG:teuthology.parallel:result is None 2023-11-15T05:53:07.360 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:53:07.360 INFO:teuthology.packaging:ref: None 2023-11-15T05:53:07.360 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:53:07.361 INFO:teuthology.packaging:branch: None 2023-11-15T05:53:07.361 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:53:07.568 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:53:07.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:53:07.800 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T05:53:07.830 INFO:teuthology.orchestra.run.smithi003.stdout:16.2.4-0.el8 2023-11-15T05:53:07.831 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T05:53:07.831 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T05:53:07.832 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T05:53:07.832 INFO:teuthology.packaging:ref: None 2023-11-15T05:53:07.833 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T05:53:07.833 INFO:teuthology.packaging:branch: None 2023-11-15T05:53:07.833 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T05:53:08.085 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:53:08.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T05:53:08.336 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T05:53:08.364 INFO:teuthology.orchestra.run.smithi137.stdout:16.2.4-0.el8 2023-11-15T05:53:08.364 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T05:53:08.365 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T05:53:08.365 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T05:53:08.366 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:53:08.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T05:53:08.404 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:53:08.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T05:53:08.439 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T05:53:08.439 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:53:08.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T05:53:08.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T05:53:08.566 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:53:08.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T05:53:08.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T05:53:08.682 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T05:53:08.683 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:53:08.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T05:53:08.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T05:53:08.795 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:53:08.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T05:53:08.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T05:53:08.901 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T05:53:08.902 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:53:08.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T05:53:08.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T05:53:09.018 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:53:09.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T05:53:09.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T05:53:09.125 INFO:teuthology.run_tasks:Running task print... 2023-11-15T05:53:09.155 INFO:teuthology.task.print:**** done install task... 2023-11-15T05:53:09.155 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T05:53:09.272 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T05:53:09.272 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T05:53:09.273 INFO:tasks.cephadm:Cluster fsid is 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:09.273 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T05:53:09.274 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T05:53:09.274 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi003': '172.21.15.3', 'mon.smithi137': '172.21.15.137'} 2023-11-15T05:53:09.274 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T05:53:09.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2023-11-15T05:53:09.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2023-11-15T05:53:09.343 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T05:53:09.344 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.489 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 05:53 /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.490 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.627 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 05:53 /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.628 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.652 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T05:53:09.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2023-11-15T05:53:09.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2023-11-15T05:53:09.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2023-11-15T05:53:09.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2023-11-15T05:53:09.831 INFO:tasks.cephadm:Writing seed config... 2023-11-15T05:53:09.832 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T05:53:09.832 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T05:53:09.833 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T05:53:09.833 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T05:53:09.833 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T05:53:09.834 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T05:53:09.834 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T05:53:09.834 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T05:53:09.834 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T05:53:09.835 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T05:53:09.835 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T05:53:09.835 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T05:53:09.836 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T05:53:09.836 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T05:53:09.836 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T05:53:09.836 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T05:53:09.837 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T05:53:09.837 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T05:53:09.837 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T05:53:09.837 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T05:53:09.838 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T05:53:09.838 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T05:53:09.839 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T05:53:09.840 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T05:53:09.840 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T05:53:09.840 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T05:53:09.841 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T05:53:09.841 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T05:53:09.841 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T05:53:09.841 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T05:53:09.842 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T05:53:09.842 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T05:53:09.842 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T05:53:09.842 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T05:53:09.843 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T05:53:09.843 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T05:53:09.843 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T05:53:09.843 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T05:53:09.844 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:53:09.845 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T05:53:09.868 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 412cc4c0-837b-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T05:53:09.869 DEBUG:teuthology.orchestra.run.smithi003:mon.smithi003> sudo journalctl -f -n 0 -u ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service 2023-11-15T05:53:09.911 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T05:53:09.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.3 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:53:09.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:-- Logs begin at Wed 2023-11-15 05:39:55 UTC. -- 2023-11-15T05:53:10.202 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:53:11.110 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-15T05:53:11.118 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying podman|docker is present... 2023-11-15T05:53:11.118 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:53:11.143 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-15T05:53:11.153 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying lvm2 is present... 2023-11-15T05:53:11.154 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying time synchronization is in place... 2023-11-15T05:53:11.154 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T05:53:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T05:53:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chrony.service 2023-11-15T05:53:11.169 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:53:11.170 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T05:53:11.176 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: enabled 2023-11-15T05:53:11.177 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T05:53:11.184 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: active 2023-11-15T05:53:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:Unit chronyd.service is enabled and running 2023-11-15T05:53:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:Repeating the final host check... 2023-11-15T05:53:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T05:53:11.208 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-15T05:53:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:podman|docker (/bin/podman) is present 2023-11-15T05:53:11.217 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl is present 2023-11-15T05:53:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:lvcreate is present 2023-11-15T05:53:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T05:53:11.223 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T05:53:11.224 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chrony.service 2023-11-15T05:53:11.230 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:53:11.231 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T05:53:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: enabled 2023-11-15T05:53:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T05:53:11.244 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: active 2023-11-15T05:53:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:Unit chronyd.service is enabled and running 2023-11-15T05:53:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:Host looks OK 2023-11-15T05:53:11.245 INFO:teuthology.orchestra.run.smithi003.stderr:Cluster fsid: 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:11.246 INFO:teuthology.orchestra.run.smithi003.stderr:Acquiring lock 140643497687528 on /run/cephadm/412cc4c0-837b-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:53:11.246 INFO:teuthology.orchestra.run.smithi003.stderr:Lock 140643497687528 acquired on /run/cephadm/412cc4c0-837b-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:53:11.246 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying IP 172.21.15.3 port 3300 ... 2023-11-15T05:53:11.247 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying IP 172.21.15.3 port 6789 ... 2023-11-15T05:53:11.247 INFO:teuthology.orchestra.run.smithi003.stderr:Base mon IP is 172.21.15.3, final addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-11-15T05:53:11.247 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip route ls 2023-11-15T05:53:11.249 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2023-11-15T05:53:11.249 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2023-11-15T05:53:11.250 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T05:53:11.252 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T05:53:11.253 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-15T05:53:11.253 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2023-11-15T05:53:11.254 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T05:53:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T05:53:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T05:53:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T05:53:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-15T05:53:11.256 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-11-15T05:53:11.257 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T05:53:11.257 INFO:teuthology.orchestra.run.smithi003.stderr:Mon IP 172.21.15.3 is in CIDR network 172.21.0.0/20 2023-11-15T05:53:11.257 INFO:teuthology.orchestra.run.smithi003.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T05:53:11.258 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T05:53:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T05:53:11.363 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T05:53:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Getting image source signatures 2023-11-15T05:53:17.857 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T05:53:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T05:53:30.603 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T05:53:30.652 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T05:53:31.016 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T05:53:31.026 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T05:53:31.878 INFO:teuthology.orchestra.run.smithi003.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T05:53:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T05:53:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T05:53:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T05:53:32.549 INFO:teuthology.orchestra.run.smithi003.stderr:stat: 167 167 2023-11-15T05:53:32.874 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial keys... 2023-11-15T05:53:32.874 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:53:33.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQDdXFRlATJBChAAI4+DtVNjQsh4LgWt1SYokw== 2023-11-15T05:53:33.511 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:53:33.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQDdXFRl6sp2MBAAGT8AadgVrUw66Ou1/TYeDw== 2023-11-15T05:53:34.157 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T05:53:34.440 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQDeXFRlE4TwGRAAqvwp0uFY1fbgtk1JMUk3jQ== 2023-11-15T05:53:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial monmap... 2023-11-15T05:53:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpcjqf6yww:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 --addv smithi003 [v2:172.21.15.3:3300,v1:172.21.15.3:6789] /tmp/monmap 2023-11-15T05:53:35.112 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T05:53:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:35.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T05:53:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:monmaptool for smithi003 [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T05:53:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: set fsid to 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:35.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T05:53:35.497 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:53:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:Creating mon... 2023-11-15T05:53:35.497 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z -v /tmp/ceph-tmpe4hhn96s:/tmp/keyring:z -v /tmp/ceph-tmpcjqf6yww:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi003 --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T05:53:35.774 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.772+0000 7f142acde700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T05:53:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.773+0000 7f142acde700 1 imported monmap: 2023-11-15T05:53:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T05:53:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T05:53:35.111800+0000 2023-11-15T05:53:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: created 2023-11-15T05:53:35.111800+0000 2023-11-15T05:53:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T05:53:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T05:53:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.smithi003 2023-11-15T05:53:35.777 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.773+0000 7f142acde700 0 /usr/bin/ceph-mon: set fsid to 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:35.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T05:53:35.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T05:53:35.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Compile date May 13 2021 2023-11-15T05:53:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: DB SUMMARY 2023-11-15T05:53:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi003/store.db dir, Total Num: 0, files: 2023-11-15T05:53:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi003/store.db: 2023-11-15T05:53:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T05:53:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T05:53:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T05:53:35.791 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.env: 0x5627045e81c0 2023-11-15T05:53:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.fs: Posix File System 2023-11-15T05:53:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.info_log: 0x5627065c8ec0 2023-11-15T05:53:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T05:53:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.statistics: (nil) 2023-11-15T05:53:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.use_fsync: 0 2023-11-15T05:53:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T05:53:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T05:53:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T05:53:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T05:53:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T05:53:35.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T05:53:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.db_log_dir: 2023-11-15T05:53:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi003/store.db 2023-11-15T05:53:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T05:53:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T05:53:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T05:53:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T05:53:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T05:53:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T05:53:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T05:53:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.write_buffer_manager: 0x5627065cf530 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T05:53:35.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T05:53:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T05:53:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T05:53:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T05:53:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T05:53:35.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T05:53:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T05:53:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.unordered_write: 0 2023-11-15T05:53:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T05:53:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T05:53:35.799 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.row_cache: None 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.wal_filter: None 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T05:53:35.800 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T05:53:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T05:53:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T05:53:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T05:53:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T05:53:35.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T05:53:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T05:53:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T05:53:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T05:53:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T05:53:35.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T05:53:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T05:53:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T05:53:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T05:53:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T05:53:35.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T05:53:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T05:53:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T05:53:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T05:53:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.max_open_files: -1 2023-11-15T05:53:35.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T05:53:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T05:53:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T05:53:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T05:53:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Compression algorithms supported: 2023-11-15T05:53:35.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T05:53:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kZSTD supported: 0 2023-11-15T05:53:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T05:53:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T05:53:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T05:53:35.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T05:53:35.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T05:53:35.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T05:53:35.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.774+0000 7f142acde700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T05:53:35.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T05:53:35.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000001 2023-11-15T05:53:35.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T05:53:35.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.merge_operator: 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_filter: None 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T05:53:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5627064e0120) 2023-11-15T05:53:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T05:53:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T05:53:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T05:53:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T05:53:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T05:53:35.812 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T05:53:35.813 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache: 0x562706516d10 2023-11-15T05:53:35.813 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T05:53:35.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T05:53:35.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T05:53:35.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T05:53:35.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T05:53:35.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T05:53:35.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T05:53:35.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T05:53:35.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T05:53:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T05:53:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T05:53:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T05:53:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T05:53:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T05:53:35.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T05:53:35.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T05:53:35.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T05:53:35.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T05:53:35.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T05:53:35.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T05:53:35.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T05:53:35.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T05:53:35.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T05:53:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T05:53:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression: NoCompression 2023-11-15T05:53:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T05:53:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T05:53:35.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T05:53:35.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.num_levels: 7 2023-11-15T05:53:35.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T05:53:35.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T05:53:35.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T05:53:35.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T05:53:35.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T05:53:35.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T05:53:35.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T05:53:35.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:53:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T05:53:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T05:53:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T05:53:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T05:53:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T05:53:35.839 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T05:53:35.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T05:53:35.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T05:53:35.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T05:53:35.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T05:53:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T05:53:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T05:53:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T05:53:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T05:53:35.841 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T05:53:35.842 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T05:53:35.842 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T05:53:35.842 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T05:53:35.842 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T05:53:35.842 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T05:53:35.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T05:53:35.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T05:53:35.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T05:53:35.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T05:53:35.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T05:53:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T05:53:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T05:53:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T05:53:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T05:53:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T05:53:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T05:53:35.846 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T05:53:35.846 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T05:53:35.846 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T05:53:35.846 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T05:53:35.846 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T05:53:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.ttl: 2592000 2023-11-15T05:53:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.775+0000 7f142acde700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T05:53:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.776+0000 7f142acde700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T05:53:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.776+0000 7f142acde700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T05:53:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.777+0000 7f142acde700 4 rocksdb: DB pointer 0x5627065db800 2023-11-15T05:53:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.777+0000 7f1413b77700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T05:53:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.777+0000 7f1413b77700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T05:53:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T05:53:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T05:53:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T05:53:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:53:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:35.852 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:35.852 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:53:35.852 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:53:35.852 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.852 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:53:35.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:35.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:35.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:35.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:35.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:35.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:53:35.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:53:35.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:53:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:53:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:53:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:53:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T05:53:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:53:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:53:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:53:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:53:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:53:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-15T05:53:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.779+0000 7f142acde700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T05:53:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.779+0000 7f142acde700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T05:53:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-15T05:53:35.779+0000 7f142acde700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi003 for mon.smithi003 2023-11-15T05:53:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:create mon.smithi003 on 2023-11-15T05:53:36.136 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:36.143 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl disable ceph.target 2023-11-15T05:53:36.151 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T05:53:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph.target 2023-11-15T05:53:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T05:53:36.348 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph.target 2023-11-15T05:53:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target 2023-11-15T05:53:36.362 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target. 2023-11-15T05:53:36.363 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target. 2023-11-15T05:53:36.453 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target 2023-11-15T05:53:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl daemon-reload 2023-11-15T05:53:36.561 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T05:53:36.590 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl reset-failed ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T05:53:36.596 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to reset failed state of unit ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service: Unit ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service not loaded. 2023-11-15T05:53:36.597 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T05:53:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target.wants/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service → /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T05:53:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T05:53:36.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:36 smithi003 systemd[1]: Starting Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7... 2023-11-15T05:53:37.045 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:53:37.057 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-15T05:53:37.058 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:53:37.067 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:53:37.068 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-15T05:53:37.068 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T05:53:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T05:53:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mon to start... 2023-11-15T05:53:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mon... 2023-11-15T05:53:37.069 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T05:53:37.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:37 smithi003 bash[111951]: 25bedf9987d1e2aede511675beb47ba282cae23c55f42137020d0af0f6ea0b05 2023-11-15T05:53:37.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:37 smithi003 systemd[1]: Started Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7. 2023-11-15T05:53:37.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.087898+0000 mon.smithi003 (mon.0) 0 : [INF] mkfs 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:37.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.070431+0000 mon.smithi003 (mon.0) 1 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-15T05:53:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.479+0000 7fa194f05700 1 Processor -- start 2023-11-15T05:53:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.479+0000 7fa194f05700 1 -- start start 2023-11-15T05:53:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.479+0000 7fa194f05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:37.484 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.479+0000 7fa194f05700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900fc5b0 con 0x7fa1900fbca0 2023-11-15T05:53:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.480+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.480+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39900/0 (socket says 172.21.15.3:39900) 2023-11-15T05:53:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.480+0000 7fa18e59c700 1 -- 172.21.15.3:0/2465510302 learned_addr learned my addr 172.21.15.3:0/2465510302 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa18e59c700 1 -- 172.21.15.3:0/2465510302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1900fc6f0 con 0x7fa1900fbca0 2023-11-15T05:53:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa18e59c700 1 --2- 172.21.15.3:0/2465510302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa18000b5f0 tx=0x7fa180006ee0).ready entity=mon.0 client_cookie=67168e5cd7602e39 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa18d59a700 1 -- 172.21.15.3:0/2465510302 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa18000d600 con 0x7fa1900fbca0 2023-11-15T05:53:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa18d59a700 1 -- 172.21.15.3:0/2465510302 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18000d760 con 0x7fa1900fbca0 2023-11-15T05:53:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa18d59a700 1 -- 172.21.15.3:0/2465510302 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa18000da30 con 0x7fa1900fbca0 2023-11-15T05:53:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.481+0000 7fa194f05700 1 -- 172.21.15.3:0/2465510302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 msgr2=0x7fa1900fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 --2- 172.21.15.3:0/2465510302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa18000b5f0 tx=0x7fa180006ee0).stop 2023-11-15T05:53:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 -- 172.21.15.3:0/2465510302 shutdown_connections 2023-11-15T05:53:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 --2- 172.21.15.3:0/2465510302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa1900fc0c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 -- 172.21.15.3:0/2465510302 >> 172.21.15.3:0/2465510302 conn(0x7fa1900f74c0 msgr2=0x7fa1900f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 -- 172.21.15.3:0/2465510302 shutdown_connections 2023-11-15T05:53:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 -- 172.21.15.3:0/2465510302 wait complete. 2023-11-15T05:53:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 Processor -- start 2023-11-15T05:53:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.482+0000 7fa194f05700 1 -- start start 2023-11-15T05:53:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa194f05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa194f05700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19010a760 con 0x7fa1900fbca0 2023-11-15T05:53:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa18e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39902/0 (socket says 172.21.15.3:39902) 2023-11-15T05:53:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa18e59c700 1 -- 172.21.15.3:0/3830689584 learned_addr learned my addr 172.21.15.3:0/3830689584 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa18e59c700 1 -- 172.21.15.3:0/3830689584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18000b200 con 0x7fa1900fbca0 2023-11-15T05:53:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa18e59c700 1 --2- 172.21.15.3:0/3830689584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa180006c70 tx=0x7fa180016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa180011b40 con 0x7fa1900fbca0 2023-11-15T05:53:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.483+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180011ca0 con 0x7fa1900fbca0 2023-11-15T05:53:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.484+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa18001d430 con 0x7fa1900fbca0 2023-11-15T05:53:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.484+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19010a960 con 0x7fa1900fbca0 2023-11-15T05:53:37.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.484+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19010ad70 con 0x7fa1900fbca0 2023-11-15T05:53:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.484+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa18000f070 con 0x7fa1900fbca0 2023-11-15T05:53:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.484+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa180027460 con 0x7fa1900fbca0 2023-11-15T05:53:37.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.485+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa190045ad0 con 0x7fa1900fbca0 2023-11-15T05:53:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.487+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa180016070 con 0x7fa1900fbca0 2023-11-15T05:53:37.562 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.560+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa190103d10 con 0x7fa1900fbca0 2023-11-15T05:53:37.562 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa17effd700 1 -- 172.21.15.3:0/3830689584 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+323 (secure 0 0 0) 0x7fa180035160 con 0x7fa1900fbca0 2023-11-15T05:53:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: cluster: 2023-11-15T05:53:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: id: 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T05:53:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: services: 2023-11-15T05:53:37.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi003 (age 0.4751s) 2023-11-15T05:53:37.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T05:53:37.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T05:53:37.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: data: 2023-11-15T05:53:37.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T05:53:37.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: pgs: 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 msgr2=0x7fa19010a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 --2- 172.21.15.3:0/3830689584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa180006c70 tx=0x7fa180016000).stop 2023-11-15T05:53:37.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 shutdown_connections 2023-11-15T05:53:37.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 --2- 172.21.15.3:0/3830689584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1900fbca0 0x7fa19010a270 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:37.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 >> 172.21.15.3:0/3830689584 conn(0x7fa1900f74c0 msgr2=0x7fa1900fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:37.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.561+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 shutdown_connections 2023-11-15T05:53:37.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:37.562+0000 7fa194f05700 1 -- 172.21.15.3:0/3830689584 wait complete. 2023-11-15T05:53:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:mon is available 2023-11-15T05:53:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T05:53:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z -v /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi003/config 2023-11-15T05:53:38.148 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.086496+0000 mon.smithi003 (mon.0) 2 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-15T05:53:38.148 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.086759+0000 mon.smithi003 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi003=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} 2023-11-15T05:53:38.148 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.090366+0000 mon.smithi003 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T05:53:38.148 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.093600+0000 mon.smithi003 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T05:53:38.149 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: cluster 2023-11-15T05:53:37.094089+0000 mon.smithi003 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T05:53:38.149 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: audit 2023-11-15T05:53:37.561557+0000 mon.smithi003 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.3:0/3830689584' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T05:53:38.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67cd4a4700 1 Processor -- start 2023-11-15T05:53:38.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67cd4a4700 1 -- start start 2023-11-15T05:53:38.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67cd4a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:38.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67cd4a4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80fcbb0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:38.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39904/0 (socket says 172.21.15.3:39904) 2023-11-15T05:53:38.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.408+0000 7f67c6ffd700 1 -- 172.21.15.3:0/4142052732 learned_addr learned my addr 172.21.15.3:0/4142052732 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:38.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.409+0000 7f67c6ffd700 1 -- 172.21.15.3:0/4142052732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80fccf0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.409+0000 7f67c6ffd700 1 --2- 172.21.15.3:0/4142052732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f67bc00b5f0 tx=0x7f67bc006ee0).ready entity=mon.0 client_cookie=2652565135f91105 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:38.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.409+0000 7f67c5ffb700 1 -- 172.21.15.3:0/4142052732 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc00d600 con 0x7f67c80fc2a0 2023-11-15T05:53:38.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.409+0000 7f67c5ffb700 1 -- 172.21.15.3:0/4142052732 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc00d760 con 0x7f67c80fc2a0 2023-11-15T05:53:38.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.409+0000 7f67c5ffb700 1 -- 172.21.15.3:0/4142052732 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc00da30 con 0x7f67c80fc2a0 2023-11-15T05:53:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 -- 172.21.15.3:0/4142052732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 msgr2=0x7f67c80fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 --2- 172.21.15.3:0/4142052732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f67bc00b5f0 tx=0x7f67bc006ee0).stop 2023-11-15T05:53:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 -- 172.21.15.3:0/4142052732 shutdown_connections 2023-11-15T05:53:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 --2- 172.21.15.3:0/4142052732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c80fc6c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:38.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 -- 172.21.15.3:0/4142052732 >> 172.21.15.3:0/4142052732 conn(0x7f67c80f5b10 msgr2=0x7f67c80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:38.418 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 -- 172.21.15.3:0/4142052732 shutdown_connections 2023-11-15T05:53:38.418 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 -- 172.21.15.3:0/4142052732 wait complete. 2023-11-15T05:53:38.418 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.410+0000 7f67cd4a4700 1 Processor -- start 2023-11-15T05:53:38.418 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67cd4a4700 1 -- start start 2023-11-15T05:53:38.419 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67cd4a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:38.419 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67cd4a4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c810c940 con 0x7f67c80fc2a0 2023-11-15T05:53:38.419 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:38.419 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39910/0 (socket says 172.21.15.3:39910) 2023-11-15T05:53:38.419 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67c6ffd700 1 -- 172.21.15.3:0/3817362184 learned_addr learned my addr 172.21.15.3:0/3817362184 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:38.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67c6ffd700 1 -- 172.21.15.3:0/3817362184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc00b200 con 0x7f67c80fc2a0 2023-11-15T05:53:38.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67c6ffd700 1 --2- 172.21.15.3:0/3817362184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f67bc006c70 tx=0x7f67bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:38.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc011b40 con 0x7f67c80fc2a0 2023-11-15T05:53:38.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.411+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc011ca0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.412+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c810cb40 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.412+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67bc01d430 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.412+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c810cf50 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.412+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f67bc00f070 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.412+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f67bc027460 con 0x7f67c80fc2a0 2023-11-15T05:53:38.421 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.413+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c8045ad0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.422 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.416+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f67bc016070 con 0x7f67c80fc2a0 2023-11-15T05:53:38.488 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.487+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f67c81015c0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.492 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.491+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f67bc01d620 con 0x7f67c80fc2a0 2023-11-15T05:53:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.491+0000 7f67af7fe700 1 -- 172.21.15.3:0/3817362184 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+462 (secure 0 0 0) 0x7f67bc0308f0 con 0x7f67c80fc2a0 2023-11-15T05:53:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [global] 2023-11-15T05:53:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: fsid = 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:38.493 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-11-15T05:53:38.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T05:53:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T05:53:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T05:53:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T05:53:38.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [mgr] 2023-11-15T05:53:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T05:53:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [osd] 2023-11-15T05:53:38.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T05:53:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T05:53:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T05:53:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T05:53:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T05:53:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T05:53:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 msgr2=0x7f67c810c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 --2- 172.21.15.3:0/3817362184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f67bc006c70 tx=0x7f67bc016000).stop 2023-11-15T05:53:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 shutdown_connections 2023-11-15T05:53:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 --2- 172.21.15.3:0/3817362184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67c80fc2a0 0x7f67c810c450 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:38.499 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 >> 172.21.15.3:0/3817362184 conn(0x7f67c80f5b10 msgr2=0x7f67c80f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:38.500 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 shutdown_connections 2023-11-15T05:53:38.500 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:38.492+0000 7f67cd4a4700 1 -- 172.21.15.3:0/3817362184 wait complete. 2023-11-15T05:53:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:Generating new minimal ceph.conf... 2023-11-15T05:53:38.851 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z -v /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi003/config 2023-11-15T05:53:39.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.221+0000 7fd75e326700 1 Processor -- start 2023-11-15T05:53:39.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.221+0000 7fd75e326700 1 -- start start 2023-11-15T05:53:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd75e326700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd75e326700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580fc750 con 0x7fd7580703b0 2023-11-15T05:53:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd757fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd757fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39926/0 (socket says 172.21.15.3:39926) 2023-11-15T05:53:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd757fff700 1 -- 172.21.15.3:0/1540341926 learned_addr learned my addr 172.21.15.3:0/1540341926 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:39.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd757fff700 1 -- 172.21.15.3:0/1540341926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758101250 con 0x7fd7580703b0 2023-11-15T05:53:39.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd757fff700 1 --2- 172.21.15.3:0/1540341926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd74800b5f0 tx=0x7fd748006ee0).ready entity=mon.0 client_cookie=88ce2f190db5c019 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:39.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.222+0000 7fd756ffd700 1 -- 172.21.15.3:0/1540341926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74800d600 con 0x7fd7580703b0 2023-11-15T05:53:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd756ffd700 1 -- 172.21.15.3:0/1540341926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd74800dbd0 con 0x7fd7580703b0 2023-11-15T05:53:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd756ffd700 1 -- 172.21.15.3:0/1540341926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd748011c00 con 0x7fd7580703b0 2023-11-15T05:53:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 -- 172.21.15.3:0/1540341926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 msgr2=0x7fd758100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 --2- 172.21.15.3:0/1540341926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd74800b5f0 tx=0x7fd748006ee0).stop 2023-11-15T05:53:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 -- 172.21.15.3:0/1540341926 shutdown_connections 2023-11-15T05:53:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 --2- 172.21.15.3:0/1540341926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd758100d60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 -- 172.21.15.3:0/1540341926 >> 172.21.15.3:0/1540341926 conn(0x7fd7580f63c0 msgr2=0x7fd7580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 -- 172.21.15.3:0/1540341926 shutdown_connections 2023-11-15T05:53:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.223+0000 7fd75e326700 1 -- 172.21.15.3:0/1540341926 wait complete. 2023-11-15T05:53:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd75e326700 1 Processor -- start 2023-11-15T05:53:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd75e326700 1 -- start start 2023-11-15T05:53:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd75e326700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd75e326700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75810d100 con 0x7fd7580703b0 2023-11-15T05:53:39.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd757fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:39.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd757fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39940/0 (socket says 172.21.15.3:39940) 2023-11-15T05:53:39.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.224+0000 7fd757fff700 1 -- 172.21.15.3:0/3666341976 learned_addr learned my addr 172.21.15.3:0/3666341976 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd757fff700 1 -- 172.21.15.3:0/3666341976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74800b200 con 0x7fd7580703b0 2023-11-15T05:53:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd757fff700 1 --2- 172.21.15.3:0/3666341976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd74800b190 tx=0x7fd748006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74801b410 con 0x7fd7580703b0 2023-11-15T05:53:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75810d300 con 0x7fd7580703b0 2023-11-15T05:53:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd74801b9e0 con 0x7fd7580703b0 2023-11-15T05:53:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd74801a880 con 0x7fd7580703b0 2023-11-15T05:53:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.225+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75810d710 con 0x7fd7580703b0 2023-11-15T05:53:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.226+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd74800f070 con 0x7fd7580703b0 2023-11-15T05:53:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.226+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd7480257d0 con 0x7fd7580703b0 2023-11-15T05:53:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.226+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd758045ad0 con 0x7fd7580703b0 2023-11-15T05:53:39.248 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.229+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd748016070 con 0x7fd7580703b0 2023-11-15T05:53:39.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.299+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd758106ad0 con 0x7fd7580703b0 2023-11-15T05:53:39.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.299+0000 7fd754ff9700 1 -- 172.21.15.3:0/3666341976 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7fd74801a9e0 con 0x7fd7580703b0 2023-11-15T05:53:39.302 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 msgr2=0x7fd75810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:39.302 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 --2- 172.21.15.3:0/3666341976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd74800b190 tx=0x7fd748006ee0).stop 2023-11-15T05:53:39.303 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 shutdown_connections 2023-11-15T05:53:39.303 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 --2- 172.21.15.3:0/3666341976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7580703b0 0x7fd75810cc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:39.303 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 >> 172.21.15.3:0/3666341976 conn(0x7fd7580f63c0 msgr2=0x7fd7580f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:39.303 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 shutdown_connections 2023-11-15T05:53:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:39.300+0000 7fd75e326700 1 -- 172.21.15.3:0/3666341976 wait complete. 2023-11-15T05:53:39.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: audit 2023-11-15T05:53:38.488428+0000 mon.smithi003 (mon.0) 8 : audit [INF] from='client.? 172.21.15.3:0/3817362184' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T05:53:39.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: audit 2023-11-15T05:53:38.491859+0000 mon.smithi003 (mon.0) 9 : audit [INF] from='client.? 172.21.15.3:0/3817362184' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T05:53:39.642 INFO:teuthology.orchestra.run.smithi003.stderr:Restarting the monitor... 2023-11-15T05:53:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl restart ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T05:53:39.732 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:39 smithi003 systemd[1]: Stopping Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7... 2023-11-15T05:53:40.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: debug 2023-11-15T05:53:39.731+0000 7f7f50fe1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi003 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T05:53:40.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[111965]: debug 2023-11-15T05:53:39.731+0000 7f7f50fe1700 -1 mon.smithi003@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T05:53:40.378 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 podman[112291]: ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003 2023-11-15T05:53:40.379 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 systemd[1]: ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service: Succeeded. 2023-11-15T05:53:40.379 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 systemd[1]: Stopped Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7. 2023-11-15T05:53:40.379 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 systemd[1]: Starting Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7... 2023-11-15T05:53:40.451 INFO:teuthology.orchestra.run.smithi003.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T05:53:40.452 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T05:53:40.826 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.824+0000 7f0c2093e700 1 Processor -- start 2023-11-15T05:53:40.827 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.824+0000 7f0c2093e700 1 -- start start 2023-11-15T05:53:40.827 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.824+0000 7f0c2093e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:40.827 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.825+0000 7f0c2093e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c0fee10 con 0x7f0c1c0703f0 2023-11-15T05:53:40.827 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.825+0000 7f0c1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:40.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.825+0000 7f0c1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39950/0 (socket says 172.21.15.3:39950) 2023-11-15T05:53:40.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.825+0000 7f0c1a59c700 1 -- 172.21.15.3:0/3179965604 learned_addr learned my addr 172.21.15.3:0/3179965604 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:40.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.825+0000 7f0c1a59c700 1 -- 172.21.15.3:0/3179965604 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c0fef50 con 0x7f0c1c0703f0 2023-11-15T05:53:40.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.826+0000 7f0c1a59c700 1 --2- 172.21.15.3:0/3179965604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0c0c00b5f0 tx=0x7f0c0c006ee0).ready entity=mon.0 client_cookie=a405cd14e1a17adc server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:40.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.826+0000 7f0c19d9b700 1 -- 172.21.15.3:0/3179965604 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c00d600 con 0x7f0c1c0703f0 2023-11-15T05:53:40.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.826+0000 7f0c19d9b700 1 -- 172.21.15.3:0/3179965604 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c0c00dbd0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.831 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.826+0000 7f0c19d9b700 1 -- 172.21.15.3:0/3179965604 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c011c00 con 0x7f0c1c0703f0 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 -- 172.21.15.3:0/3179965604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 msgr2=0x7f0c1c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 --2- 172.21.15.3:0/3179965604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0c0c00b5f0 tx=0x7f0c0c006ee0).stop 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 -- 172.21.15.3:0/3179965604 shutdown_connections 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 --2- 172.21.15.3:0/3179965604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c0fe920 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 -- 172.21.15.3:0/3179965604 >> 172.21.15.3:0/3179965604 conn(0x7f0c1c0f63c0 msgr2=0x7f0c1c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:40.832 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 -- 172.21.15.3:0/3179965604 shutdown_connections 2023-11-15T05:53:40.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.827+0000 7f0c2093e700 1 -- 172.21.15.3:0/3179965604 wait complete. 2023-11-15T05:53:40.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c2093e700 1 Processor -- start 2023-11-15T05:53:40.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c2093e700 1 -- start start 2023-11-15T05:53:40.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c2093e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:40.833 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c2093e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c10d1d0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39952/0 (socket says 172.21.15.3:39952) 2023-11-15T05:53:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c1a59c700 1 -- 172.21.15.3:0/2723723948 learned_addr learned my addr 172.21.15.3:0/2723723948 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c1a59c700 1 -- 172.21.15.3:0/2723723948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c00b200 con 0x7f0c1c0703f0 2023-11-15T05:53:40.834 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.828+0000 7f0c1a59c700 1 --2- 172.21.15.3:0/2723723948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0c0c00e040 tx=0x7f0c0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:40.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c00d9f0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c0c01a9f0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c10d3d0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.835 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c01b8c0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c10d7e0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.836 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0c0c00f070 con 0x7f0c1c0703f0 2023-11-15T05:53:40.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.829+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0c0c025ae0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.830+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c045ad0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.837 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.833+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0c0c016070 con 0x7f0c1c0703f0 2023-11-15T05:53:40.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.902+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0c1c10e6e0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.908+0000 7f0c137fe700 1 -- 172.21.15.3:0/2723723948 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0c0c01bdc0 con 0x7f0c1c0703f0 2023-11-15T05:53:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 msgr2=0x7f0c1c10cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 --2- 172.21.15.3:0/2723723948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0c0c00e040 tx=0x7f0c0c006ee0).stop 2023-11-15T05:53:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 shutdown_connections 2023-11-15T05:53:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 --2- 172.21.15.3:0/2723723948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c1c0703f0 0x7f0c1c10cce0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 >> 172.21.15.3:0/2723723948 conn(0x7f0c1c0f63c0 msgr2=0x7f0c1c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:40.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 shutdown_connections 2023-11-15T05:53:40.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:40.909+0000 7f0c2093e700 1 -- 172.21.15.3:0/2723723948 wait complete. 2023-11-15T05:53:40.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 bash[112371]: 10691c46bdd2010c148eae45f84361198fcc2772562f4521da1b9d4fc192b6e3 2023-11-15T05:53:40.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 systemd[1]: Started Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7. 2023-11-15T05:53:40.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.456+0000 7f9112355700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T05:53:40.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.456+0000 7f9112355700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T05:53:40.934 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.456+0000 7f9112355700 0 pidfile_write: ignore empty --pid-file 2023-11-15T05:53:40.934 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.462+0000 7f9112355700 0 load: jerasure load: lrc load: isa 2023-11-15T05:53:40.934 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T05:53:40.935 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.935 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T05:53:40.938 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Compile date May 13 2021 2023-11-15T05:53:40.939 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: DB SUMMARY 2023-11-15T05:53:40.939 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.939 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T05:53:40.939 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.939 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T05:53:40.940 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.940 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T05:53:40.940 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.940 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi003/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T05:53:40.940 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.941 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi003/store.db: 000006.log size: 92898 ; 2023-11-15T05:53:40.941 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.941 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T05:53:40.941 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T05:53:40.941 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T05:53:40.942 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.env: 0x55cdb0c241c0 2023-11-15T05:53:40.942 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.fs: Posix File System 2023-11-15T05:53:40.942 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.info_log: 0x55cdb3311960 2023-11-15T05:53:40.942 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T05:53:40.943 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.statistics: (nil) 2023-11-15T05:53:40.943 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.use_fsync: 0 2023-11-15T05:53:40.943 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T05:53:40.943 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T05:53:40.944 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T05:53:40.944 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T05:53:40.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T05:53:40.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T05:53:40.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T05:53:40.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T05:53:40.945 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T05:53:40.946 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T05:53:40.946 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T05:53:40.946 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.db_log_dir: 2023-11-15T05:53:40.946 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi003/store.db 2023-11-15T05:53:40.947 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T05:53:40.947 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T05:53:40.947 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T05:53:40.947 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T05:53:40.947 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T05:53:40.948 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T05:53:40.948 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T05:53:40.948 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T05:53:40.948 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T05:53:40.949 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T05:53:40.949 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.write_buffer_manager: 0x55cdb3f86030 2023-11-15T05:53:40.949 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T05:53:40.949 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T05:53:40.950 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T05:53:40.950 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T05:53:40.950 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T05:53:40.950 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T05:53:40.950 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T05:53:40.951 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T05:53:40.951 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T05:53:40.951 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.unordered_write: 0 2023-11-15T05:53:40.951 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T05:53:40.951 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T05:53:40.952 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T05:53:40.952 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T05:53:40.952 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.row_cache: None 2023-11-15T05:53:40.952 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.wal_filter: None 2023-11-15T05:53:40.952 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T05:53:40.953 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T05:53:40.953 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T05:53:40.953 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T05:53:40.953 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T05:53:40.954 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T05:53:40.954 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T05:53:40.955 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T05:53:40.955 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T05:53:40.955 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T05:53:40.956 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T05:53:40.956 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T05:53:40.956 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T05:53:40.956 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T05:53:40.956 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T05:53:40.957 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T05:53:40.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.max_open_files: -1 2023-11-15T05:53:40.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T05:53:40.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T05:53:40.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T05:53:40.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T05:53:40.960 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Compression algorithms supported: 2023-11-15T05:53:40.960 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T05:53:40.960 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kZSTD supported: 0 2023-11-15T05:53:40.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T05:53:40.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T05:53:40.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T05:53:40.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T05:53:40.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T05:53:40.962 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T05:53:40.962 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.463+0000 7f9112355700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T05:53:40.962 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000005 2023-11-15T05:53:40.962 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.962 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T05:53:40.963 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.963 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T05:53:40.963 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.merge_operator: 2023-11-15T05:53:40.963 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_filter: None 2023-11-15T05:53:40.963 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T05:53:40.964 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T05:53:40.964 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T05:53:40.964 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cdb3228128) 2023-11-15T05:53:40.964 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cache_index_and_filter_blocks: 1 2023-11-15T05:53:40.965 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T05:53:40.965 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T05:53:40.965 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: pin_top_level_index_and_filter: 1 2023-11-15T05:53:40.965 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: index_type: 0 2023-11-15T05:53:40.966 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: data_block_index_type: 0 2023-11-15T05:53:40.966 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: index_shortening: 1 2023-11-15T05:53:40.966 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T05:53:40.966 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: hash_index_allow_collision: 1 2023-11-15T05:53:40.966 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: checksum: 1 2023-11-15T05:53:40.967 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: no_block_cache: 0 2023-11-15T05:53:40.967 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_cache: 0x55cdb325ef10 2023-11-15T05:53:40.967 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_cache_name: BinnedLRUCache 2023-11-15T05:53:40.968 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_cache_options: 2023-11-15T05:53:40.968 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: capacity : 536870912 2023-11-15T05:53:40.968 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: num_shard_bits : 4 2023-11-15T05:53:40.968 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: strict_capacity_limit : 0 2023-11-15T05:53:40.968 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: high_pri_pool_ratio: 0.000 2023-11-15T05:53:40.969 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_cache_compressed: (nil) 2023-11-15T05:53:40.969 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: persistent_cache: (nil) 2023-11-15T05:53:40.969 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_size: 4096 2023-11-15T05:53:40.969 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_size_deviation: 10 2023-11-15T05:53:40.969 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_restart_interval: 16 2023-11-15T05:53:40.970 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: index_block_restart_interval: 1 2023-11-15T05:53:40.970 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: metadata_block_size: 4096 2023-11-15T05:53:40.970 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: partition_filters: 0 2023-11-15T05:53:40.970 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: use_delta_encoding: 1 2023-11-15T05:53:40.970 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T05:53:40.973 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: whole_key_filtering: 1 2023-11-15T05:53:40.974 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: verify_compression: 0 2023-11-15T05:53:40.975 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: read_amp_bytes_per_bit: 0 2023-11-15T05:53:40.975 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: format_version: 2 2023-11-15T05:53:40.975 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: enable_index_compression: 1 2023-11-15T05:53:40.975 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: block_align: 0 2023-11-15T05:53:40.975 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.976 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T05:53:40.976 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T05:53:40.976 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression: NoCompression 2023-11-15T05:53:40.976 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T05:53:40.976 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T05:53:40.977 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T05:53:40.977 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.num_levels: 7 2023-11-15T05:53:40.977 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T05:53:40.978 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T05:53:40.978 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T05:53:40.978 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T05:53:40.978 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T05:53:40.978 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T05:53:40.979 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T05:53:40.979 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:53:40.979 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T05:53:40.979 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T05:53:40.979 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T05:53:40.980 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T05:53:40.980 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T05:53:40.980 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T05:53:40.980 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T05:53:40.980 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T05:53:40.981 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T05:53:40.981 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T05:53:40.981 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T05:53:40.983 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T05:53:40.983 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T05:53:40.984 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T05:53:40.984 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T05:53:40.984 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T05:53:40.985 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T05:53:40.985 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T05:53:40.985 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T05:53:40.986 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T05:53:40.986 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T05:53:40.986 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T05:53:40.986 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T05:53:40.987 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T05:53:40.987 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T05:53:40.987 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T05:53:40.987 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T05:53:40.987 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T05:53:40.988 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T05:53:40.988 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T05:53:40.988 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T05:53:40.988 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T05:53:40.988 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T05:53:40.989 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T05:53:40.989 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T05:53:40.989 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T05:53:40.989 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T05:53:40.989 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T05:53:40.990 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T05:53:40.990 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T05:53:40.990 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T05:53:40.990 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T05:53:40.990 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T05:53:40.991 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T05:53:40.991 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T05:53:40.991 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T05:53:40.991 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T05:53:40.991 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T05:53:40.992 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T05:53:40.992 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T05:53:40.992 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T05:53:40.992 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.ttl: 2592000 2023-11-15T05:53:40.992 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.464+0000 7f9112355700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T05:53:40.993 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.465+0000 7f9112355700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T05:53:40.993 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.993 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.465+0000 7f9112355700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T05:53:40.993 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.993 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.465+0000 7f9112355700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700027620466045, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.465+0000 7f9112355700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.466+0000 7f9112355700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.466+0000 7f9112355700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700027620467620, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88678, "table_properties": {"data_size": 86793, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79934, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700027620, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.466+0000 7f9112355700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.994 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.467+0000 7f9112355700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700027620468317, "job": 1, "event": "recovery_finished"} 2023-11-15T05:53:40.995 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.468+0000 7f9112355700 4 rocksdb: DB pointer 0x55cdb3323800 2023-11-15T05:53:40.995 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.469+0000 7f90f916d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T05:53:40.995 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.469+0000 7f90f916d700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T05:53:40.995 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** DB Stats ** 2023-11-15T05:53:40.995 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:40.996 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T05:53:40.996 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T05:53:40.996 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:53:40.996 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T05:53:40.996 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** Compaction Stats [default] ** 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:40.997 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: L0 2/0 88.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:40.998 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Sum 2/0 88.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:40.998 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:40.998 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:40.998 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** Compaction Stats [default] ** 2023-11-15T05:53:40.998 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:40.999 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:40.999 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:40.999 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:40.999 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:40.999 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:41.000 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:53:41.000 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:53:41.001 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:53:41.001 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Cumulative compaction: 0.00 GB write, 17.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:41.001 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Interval compaction: 0.00 GB write, 17.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:41.001 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:53:41.001 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.002 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:53:41.002 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.002 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** Compaction Stats [default] ** 2023-11-15T05:53:41.002 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:41.002 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:41.003 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: L0 2/0 88.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:41.003 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Sum 2/0 88.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:41.003 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T05:53:41.003 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.003 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** Compaction Stats [default] ** 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:41.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T05:53:41.005 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T05:53:41.005 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T05:53:41.005 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T05:53:41.005 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Cumulative compaction: 0.00 GB write, 17.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:41.005 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T05:53:41.006 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T05:53:41.006 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.006 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T05:53:41.006 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.006 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.470+0000 7f9112355700 0 starting mon.smithi003 rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi003 fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.471+0000 7f9112355700 1 mon.smithi003@-1(???) e1 preinit fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).mds e1 new map 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).mds e1 print_map 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: e1 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T05:53:41.007 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T05:53:41.008 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: legacy client fscid: -1 2023-11-15T05:53:41.008 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.008 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: No filesystems configured 2023-11-15T05:53:41.008 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: 2023-11-15T05:53:41.008 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T05:53:41.009 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:53:41.009 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:53:41.009 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T05:53:41.009 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:38.488428+0000 mon.smithi003 (mon.0) 8 : audit [INF] from='client.? 172.21.15.3:0/3817362184' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T05:53:41.009 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:38.491859+0000 mon.smithi003 (mon.0) 9 : audit [INF] from='client.? 172.21.15.3:0/3817362184' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T05:53:41.010 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T05:53:40.472+0000 7f9112355700 1 mon.smithi003@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T05:53:41.010 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:40.493525+0000 mon.smithi003 (mon.0) 1 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-15T05:53:41.010 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:40.493603+0000 mon.smithi003 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi003=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} 2023-11-15T05:53:41.010 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:40.493641+0000 mon.smithi003 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T05:53:41.010 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:40.493671+0000 mon.smithi003 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T05:53:41.011 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:40.494146+0000 mon.smithi003 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T05:53:41.328 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T05:53:41.328 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:53:41.329 INFO:teuthology.orchestra.run.smithi003.stderr:Creating mgr... 2023-11-15T05:53:41.329 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying port 9283 ... 2023-11-15T05:53:41.331 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl daemon-reload 2023-11-15T05:53:41.448 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke 2023-11-15T05:53:41.475 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl reset-failed ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke 2023-11-15T05:53:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to reset failed state of unit ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke.service: Unit ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke.service not loaded. 2023-11-15T05:53:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke 2023-11-15T05:53:41.488 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7.target.wants/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke.service → /etc/systemd/system/ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T05:53:41.584 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mgr.smithi003.cldzke 2023-11-15T05:53:41.911 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:53:41.923 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-15T05:53:41.923 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:53:41.931 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:53:41.932 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-15T05:53:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T05:53:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T05:53:41.933 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:53:41.941 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-15T05:53:41.942 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:53:41.950 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:53:41.951 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-15T05:53:41.951 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T05:53:41.951 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr to start... 2023-11-15T05:53:41.951 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr... 2023-11-15T05:53:41.952 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:53:42.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:40.908777+0000 mon.smithi003 (mon.0) 6 : audit [INF] from='client.? 172.21.15.3:0/2723723948' entity='client.admin' 2023-11-15T05:53:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.364+0000 7fcff1f8b700 1 Processor -- start 2023-11-15T05:53:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.364+0000 7fcff1f8b700 1 -- start start 2023-11-15T05:53:42.366 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.364+0000 7fcff1f8b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.364+0000 7fcff1f8b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0728d0 con 0x7fcfec0724b0 2023-11-15T05:53:42.368 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.365+0000 7fcfeb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.365+0000 7fcfeb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39992/0 (socket says 172.21.15.3:39992) 2023-11-15T05:53:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.365+0000 7fcfeb7fe700 1 -- 172.21.15.3:0/2074290094 learned_addr learned my addr 172.21.15.3:0/2074290094 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.365+0000 7fcfeb7fe700 1 -- 172.21.15.3:0/2074290094 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec070f40 con 0x7fcfec0724b0 2023-11-15T05:53:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.365+0000 7fcfeb7fe700 1 --2- 172.21.15.3:0/2074290094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcfdc00b5f0 tx=0x7fcfdc006ee0).ready entity=mon.0 client_cookie=cfb074e8d4d86be1 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.366+0000 7fcfea7fc700 1 -- 172.21.15.3:0/2074290094 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfdc00d600 con 0x7fcfec0724b0 2023-11-15T05:53:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.366+0000 7fcfea7fc700 1 -- 172.21.15.3:0/2074290094 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcfdc00dbd0 con 0x7fcfec0724b0 2023-11-15T05:53:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.366+0000 7fcfea7fc700 1 -- 172.21.15.3:0/2074290094 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfdc011c00 con 0x7fcfec0724b0 2023-11-15T05:53:42.370 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.366+0000 7fcff1f8b700 1 -- 172.21.15.3:0/2074290094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 msgr2=0x7fcfec070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.366+0000 7fcff1f8b700 1 --2- 172.21.15.3:0/2074290094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fcfdc00b5f0 tx=0x7fcfdc006ee0).stop 2023-11-15T05:53:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 -- 172.21.15.3:0/2074290094 shutdown_connections 2023-11-15T05:53:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 --2- 172.21.15.3:0/2074290094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec070a50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 -- 172.21.15.3:0/2074290094 >> 172.21.15.3:0/2074290094 conn(0x7fcfec0f74c0 msgr2=0x7fcfec0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 -- 172.21.15.3:0/2074290094 shutdown_connections 2023-11-15T05:53:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 -- 172.21.15.3:0/2074290094 wait complete. 2023-11-15T05:53:42.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.367+0000 7fcff1f8b700 1 Processor -- start 2023-11-15T05:53:42.373 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcff1f8b700 1 -- start start 2023-11-15T05:53:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcff1f8b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcff1f8b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec10d1f0 con 0x7fcfec0724b0 2023-11-15T05:53:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcfeb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcfeb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40000/0 (socket says 172.21.15.3:40000) 2023-11-15T05:53:42.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.368+0000 7fcfeb7fe700 1 -- 172.21.15.3:0/1102944402 learned_addr learned my addr 172.21.15.3:0/1102944402 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfeb7fe700 1 -- 172.21.15.3:0/1102944402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfdc00b200 con 0x7fcfec0724b0 2023-11-15T05:53:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfeb7fe700 1 --2- 172.21.15.3:0/1102944402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcfdc00e040 tx=0x7fcfdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfdc00d9f0 con 0x7fcfec0724b0 2023-11-15T05:53:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec10d3f0 con 0x7fcfec0724b0 2023-11-15T05:53:42.375 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcfdc01a9f0 con 0x7fcfec0724b0 2023-11-15T05:53:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfdc01b8c0 con 0x7fcfec0724b0 2023-11-15T05:53:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec10d800 con 0x7fcfec0724b0 2023-11-15T05:53:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fcfdc00f070 con 0x7fcfec0724b0 2023-11-15T05:53:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.369+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcfdc025b30 con 0x7fcfec0724b0 2023-11-15T05:53:42.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.370+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec045ad0 con 0x7fcfec0724b0 2023-11-15T05:53:42.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.372+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fcfdc016070 con 0x7fcfec0724b0 2023-11-15T05:53:42.458 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.446+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcfec10ec50 con 0x7fcfec0724b0 2023-11-15T05:53:42.458 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.447+0000 7fcfd3fff700 1 -- 172.21.15.3:0/1102944402 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fcfdc01bdc0 con 0x7fcfec0724b0 2023-11-15T05:53:42.458 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "412cc4c0-837b-11ee-8dbe-212e2dc638e7", 2023-11-15T05:53:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:53:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:53:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:53:42.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:53:42.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:53:42.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:53:42.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-15T05:53:42.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:42.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:53:42.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-15T05:53:42.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:42.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T05:53:42.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:53:42.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:42.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:53:42.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:53:42.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:53:42.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:53:42.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:53:42.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:53:42.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:53:42.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:53:42.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:53:42.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:53:42.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:53:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:53:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:53:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:53:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:53:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:53:42.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:42.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:53:42.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:53:42.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:53:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-15T05:53:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:53:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:53:42.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:53:42.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:53:42.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:42.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:42.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:53:42.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:42.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:53:37.088356+0000", 2023-11-15T05:53:42.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:42.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:42.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:53:42.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:42.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 msgr2=0x7fcfec10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:42.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 --2- 172.21.15.3:0/1102944402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcfdc00e040 tx=0x7fcfdc006ee0).stop 2023-11-15T05:53:42.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 shutdown_connections 2023-11-15T05:53:42.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 --2- 172.21.15.3:0/1102944402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfec0724b0 0x7fcfec10cd00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:42.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 >> 172.21.15.3:0/1102944402 conn(0x7fcfec0f74c0 msgr2=0x7fcfec0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:42.480 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 shutdown_connections 2023-11-15T05:53:42.480 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:42.448+0000 7fcff1f8b700 1 -- 172.21.15.3:0/1102944402 wait complete. 2023-11-15T05:53:42.816 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (1/15)... 2023-11-15T05:53:43.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:42.447631+0000 mon.smithi003 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.3:0/1102944402' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:53:44.818 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:53:45.183 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.181+0000 7f0b07fd8700 1 Processor -- start 2023-11-15T05:53:45.183 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.181+0000 7f0b07fd8700 1 -- start start 2023-11-15T05:53:45.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.181+0000 7f0b07fd8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:45.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.181+0000 7f0b07fd8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b000fb490 con 0x7f0b000fab80 2023-11-15T05:53:45.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b05d74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b05d74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34806/0 (socket says 172.21.15.3:34806) 2023-11-15T05:53:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b05d74700 1 -- 172.21.15.3:0/2617609628 learned_addr learned my addr 172.21.15.3:0/2617609628 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b05d74700 1 -- 172.21.15.3:0/2617609628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b000fb5d0 con 0x7f0b000fab80 2023-11-15T05:53:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b05d74700 1 --2- 172.21.15.3:0/2617609628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0afc00b5f0 tx=0x7f0afc006ee0).ready entity=mon.0 client_cookie=4b4865794edb93cb server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b04d72700 1 -- 172.21.15.3:0/2617609628 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc00d600 con 0x7f0b000fab80 2023-11-15T05:53:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b04d72700 1 -- 172.21.15.3:0/2617609628 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0afc00dbd0 con 0x7f0b000fab80 2023-11-15T05:53:45.186 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.182+0000 7f0b04d72700 1 -- 172.21.15.3:0/2617609628 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc011c00 con 0x7f0b000fab80 2023-11-15T05:53:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2617609628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 msgr2=0x7f0b000fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 --2- 172.21.15.3:0/2617609628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f0afc00b5f0 tx=0x7f0afc006ee0).stop 2023-11-15T05:53:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2617609628 shutdown_connections 2023-11-15T05:53:45.187 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 --2- 172.21.15.3:0/2617609628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b000fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2617609628 >> 172.21.15.3:0/2617609628 conn(0x7f0b000f63c0 msgr2=0x7f0b000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2617609628 shutdown_connections 2023-11-15T05:53:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.183+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2617609628 wait complete. 2023-11-15T05:53:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b07fd8700 1 Processor -- start 2023-11-15T05:53:45.188 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b07fd8700 1 -- start start 2023-11-15T05:53:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b07fd8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b07fd8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b000ff290 con 0x7f0b000fab80 2023-11-15T05:53:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b05d74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:45.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b05d74700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34808/0 (socket says 172.21.15.3:34808) 2023-11-15T05:53:45.191 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.184+0000 7f0b05d74700 1 -- 172.21.15.3:0/2172568913 learned_addr learned my addr 172.21.15.3:0/2172568913 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0b05d74700 1 -- 172.21.15.3:0/2172568913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc00b200 con 0x7f0b000fab80 2023-11-15T05:53:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0b05d74700 1 --2- 172.21.15.3:0/2172568913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0afc00e040 tx=0x7f0afc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc01b410 con 0x7f0b000fab80 2023-11-15T05:53:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0afc01b9e0 con 0x7f0b000fab80 2023-11-15T05:53:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0afc01a820 con 0x7f0b000fab80 2023-11-15T05:53:45.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b000ff430 con 0x7f0b000fab80 2023-11-15T05:53:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b000ff840 con 0x7f0b000fab80 2023-11-15T05:53:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0afc00f070 con 0x7f0b000fab80 2023-11-15T05:53:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.185+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0afc025b30 con 0x7f0b000fab80 2023-11-15T05:53:45.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.186+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b00045ad0 con 0x7f0b000fab80 2023-11-15T05:53:45.195 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.188+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0afc016070 con 0x7f0b000fab80 2023-11-15T05:53:45.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.263+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0b00104960 con 0x7f0b000fab80 2023-11-15T05:53:45.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.263+0000 7f0aee7fc700 1 -- 172.21.15.3:0/2172568913 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f0afc01ad70 con 0x7f0b000fab80 2023-11-15T05:53:45.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:45.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "412cc4c0-837b-11ee-8dbe-212e2dc638e7", 2023-11-15T05:53:45.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:53:45.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:53:45.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:53:45.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:53:45.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:53:45.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:53:45.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-15T05:53:45.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:45.276 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:53:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-15T05:53:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:45.277 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T05:53:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:53:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:53:45.278 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:53:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:53:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:53:45.279 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:53:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:53:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:53:45.280 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:53:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:53:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:53:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:53:45.281 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:53:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:53:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:53:45.282 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:53:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:53:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:53:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:53:45.283 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.284 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:53:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:53:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:53:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.285 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:53:45.286 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-15T05:53:45.286 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:53:45.286 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:53:45.287 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:53:45.287 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:53:45.287 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:45.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:45.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:53:45.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:45.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:53:37.088356+0000", 2023-11-15T05:53:45.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:45.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:45.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:53:45.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:45.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 msgr2=0x7f0b00100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:45.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 --2- 172.21.15.3:0/2172568913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0afc00e040 tx=0x7f0afc006ee0).stop 2023-11-15T05:53:45.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 shutdown_connections 2023-11-15T05:53:45.292 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 --2- 172.21.15.3:0/2172568913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b000fab80 0x7f0b00100e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:45.292 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 >> 172.21.15.3:0/2172568913 conn(0x7f0b000f63c0 msgr2=0x7f0b00070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:45.292 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 shutdown_connections 2023-11-15T05:53:45.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:45.264+0000 7f0b07fd8700 1 -- 172.21.15.3:0/2172568913 wait complete. 2023-11-15T05:53:45.624 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (2/15)... 2023-11-15T05:53:45.626 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:45.263997+0000 mon.smithi003 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.3:0/2172568913' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:53:47.625 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:53:47.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:47.557648+0000 mon.smithi003 (mon.0) 9 : cluster [INF] Activating manager daemon smithi003.cldzke 2023-11-15T05:53:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:47.560508+0000 mon.smithi003 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi003.cldzke(active, starting, since 0.0029796s) 2023-11-15T05:53:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.561550+0000 mon.smithi003 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:53:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.561765+0000 mon.smithi003 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:53:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.561958+0000 mon.smithi003 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:53:47.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.562156+0000 mon.smithi003 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-15T05:53:47.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.562438+0000 mon.smithi003 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mgr metadata", "who": "smithi003.cldzke", "id": "smithi003.cldzke"}]: dispatch 2023-11-15T05:53:47.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:47.577241+0000 mon.smithi003 (mon.0) 16 : cluster [INF] Manager daemon smithi003.cldzke is now available 2023-11-15T05:53:47.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.585264+0000 mon.smithi003 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:47.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.591198+0000 mon.smithi003 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:53:47.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.592553+0000 mon.smithi003 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:53:47.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.598014+0000 mon.smithi003 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:47.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:47.600076+0000 mon.smithi003 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.3:0/710536799' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f6b58700 1 Processor -- start 2023-11-15T05:53:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f6b58700 1 -- start start 2023-11-15T05:53:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f6b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f6b58700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00fc5c0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.007 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f48f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:48.007 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.002+0000 7ff2f48f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34854/0 (socket says 172.21.15.3:34854) 2023-11-15T05:53:48.007 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2f48f4700 1 -- 172.21.15.3:0/2582405622 learned_addr learned my addr 172.21.15.3:0/2582405622 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:48.007 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2f48f4700 1 -- 172.21.15.3:0/2582405622 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00fc700 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.007 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2f48f4700 1 --2- 172.21.15.3:0/2582405622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff2e400b5f0 tx=0x7ff2e4006ee0).ready entity=mon.0 client_cookie=da234142189954f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:48.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2ef7fe700 1 -- 172.21.15.3:0/2582405622 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e400d600 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2ef7fe700 1 -- 172.21.15.3:0/2582405622 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff2e400dbd0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.003+0000 7ff2ef7fe700 1 -- 172.21.15.3:0/2582405622 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e4011c00 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 -- 172.21.15.3:0/2582405622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 msgr2=0x7ff2f00fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:48.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 --2- 172.21.15.3:0/2582405622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff2e400b5f0 tx=0x7ff2e4006ee0).stop 2023-11-15T05:53:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 -- 172.21.15.3:0/2582405622 shutdown_connections 2023-11-15T05:53:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 --2- 172.21.15.3:0/2582405622 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f00fc0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 -- 172.21.15.3:0/2582405622 >> 172.21.15.3:0/2582405622 conn(0x7ff2f00f74b0 msgr2=0x7ff2f00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 -- 172.21.15.3:0/2582405622 shutdown_connections 2023-11-15T05:53:48.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 -- 172.21.15.3:0/2582405622 wait complete. 2023-11-15T05:53:48.010 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.004+0000 7ff2f6b58700 1 Processor -- start 2023-11-15T05:53:48.010 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f6b58700 1 -- start start 2023-11-15T05:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f6b58700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f6b58700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f010d1c0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f48f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f48f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34868/0 (socket says 172.21.15.3:34868) 2023-11-15T05:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f48f4700 1 -- 172.21.15.3:0/1536514586 learned_addr learned my addr 172.21.15.3:0/1536514586 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f48f4700 1 -- 172.21.15.3:0/1536514586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e400b200 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.005+0000 7ff2f48f4700 1 --2- 172.21.15.3:0/1536514586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff2e400b1d0 tx=0x7ff2e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.006+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e401b410 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.006+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff2e401b9e0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.006+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f010d3c0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.006+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e401a840 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.006+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f010d7d0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.007+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7ff2e400f070 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.007+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff2e4040fe0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.007+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f0045ad0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.010+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff2e4016070 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.089 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.082+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff2f010ecc0 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.089 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.082+0000 7ff2ed7fa700 1 -- 172.21.15.3:0/1536514586 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7ff2e4022760 con 0x7ff2f00fbcb0 2023-11-15T05:53:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "412cc4c0-837b-11ee-8dbe-212e2dc638e7", 2023-11-15T05:53:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:53:48.090 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:53:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-15T05:53:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:53:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-15T05:53:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T05:53:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:53:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:53:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:53:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:53:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:53:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:53:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:53:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:53:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:53:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:53:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:53:48.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:53:48.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:53:48.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:53:48.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:53:48.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-15T05:53:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:53:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:53:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:53:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:53:48.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:53:37.088356+0000", 2023-11-15T05:53:48.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:53:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 msgr2=0x7ff2f010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:48.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 --2- 172.21.15.3:0/1536514586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff2e400b1d0 tx=0x7ff2e4006ee0).stop 2023-11-15T05:53:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 shutdown_connections 2023-11-15T05:53:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 --2- 172.21.15.3:0/1536514586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff2f00fbcb0 0x7ff2f010ccd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 >> 172.21.15.3:0/1536514586 conn(0x7ff2f00f74b0 msgr2=0x7ff2f00f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 shutdown_connections 2023-11-15T05:53:48.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:48.083+0000 7ff2f6b58700 1 -- 172.21.15.3:0/1536514586 wait complete. 2023-11-15T05:53:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (3/15)... 2023-11-15T05:53:48.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:48.083214+0000 mon.smithi003 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.3:0/1536514586' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:53:48.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:48.563493+0000 mon.smithi003 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi003.cldzke(active, since 1.00597s) 2023-11-15T05:53:50.417 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T05:53:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.794+0000 7f21aa876700 1 Processor -- start 2023-11-15T05:53:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21aa876700 1 -- start start 2023-11-15T05:53:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21aa876700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21aa876700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a4072940 con 0x7f21a4072520 2023-11-15T05:53:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34876/0 (socket says 172.21.15.3:34876) 2023-11-15T05:53:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21a3fff700 1 -- 172.21.15.3:0/3568455003 learned_addr learned my addr 172.21.15.3:0/3568455003 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21a3fff700 1 -- 172.21.15.3:0/3568455003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4070fb0 con 0x7f21a4072520 2023-11-15T05:53:50.801 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.795+0000 7f21a3fff700 1 --2- 172.21.15.3:0/3568455003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f219400b5f0 tx=0x7f2194006ee0).ready entity=mon.0 client_cookie=b750a0823f73f673 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21a2ffd700 1 -- 172.21.15.3:0/3568455003 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f219400d600 con 0x7f21a4072520 2023-11-15T05:53:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21a2ffd700 1 -- 172.21.15.3:0/3568455003 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f219400dbd0 con 0x7f21a4072520 2023-11-15T05:53:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21a2ffd700 1 -- 172.21.15.3:0/3568455003 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2194011c00 con 0x7f21a4072520 2023-11-15T05:53:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 -- 172.21.15.3:0/3568455003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 msgr2=0x7f21a4070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:50.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 --2- 172.21.15.3:0/3568455003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f219400b5f0 tx=0x7f2194006ee0).stop 2023-11-15T05:53:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 -- 172.21.15.3:0/3568455003 shutdown_connections 2023-11-15T05:53:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 --2- 172.21.15.3:0/3568455003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a4070ac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 -- 172.21.15.3:0/3568455003 >> 172.21.15.3:0/3568455003 conn(0x7f21a40f74b0 msgr2=0x7f21a40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.796+0000 7f21aa876700 1 -- 172.21.15.3:0/3568455003 shutdown_connections 2023-11-15T05:53:50.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.797+0000 7f21aa876700 1 -- 172.21.15.3:0/3568455003 wait complete. 2023-11-15T05:53:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.797+0000 7f21aa876700 1 Processor -- start 2023-11-15T05:53:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.797+0000 7f21aa876700 1 -- start start 2023-11-15T05:53:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21aa876700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:50.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21aa876700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a410d220 con 0x7f21a4072520 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34886/0 (socket says 172.21.15.3:34886) 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a3fff700 1 -- 172.21.15.3:0/3836340955 learned_addr learned my addr 172.21.15.3:0/3836340955 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a3fff700 1 -- 172.21.15.3:0/3836340955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219400b200 con 0x7f21a4072520 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a3fff700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f219400e040 tx=0x7f2194006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f219400d9f0 con 0x7f21a4072520 2023-11-15T05:53:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f219401a9f0 con 0x7f21a4072520 2023-11-15T05:53:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a410d420 con 0x7f21a4072520 2023-11-15T05:53:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f219401b8c0 con 0x7f21a4072520 2023-11-15T05:53:50.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.798+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a410d830 con 0x7f21a4072520 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.799+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f219400f070 con 0x7f21a4072520 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.799+0000 7f21a0ff9700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 0x7f2190032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.799+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2194041240 con 0x7f21a4072520 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.800+0000 7f21a37fe700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 0x7f2190032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.800+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a4101dd0 con 0x7f21a4072520 2023-11-15T05:53:50.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.800+0000 7f21a37fe700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 0x7f2190032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f21980096a0 tx=0x7f2198006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:50.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:50.804+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2194016030 con 0x7f21a4072520 2023-11-15T05:53:50.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:49.615212+0000 mon.smithi003 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi003.cldzke(active, since 2s) 2023-11-15T05:53:51.045 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.033+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f21a4106c30 con 0x7f21a4072520 2023-11-15T05:53:51.046 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.034+0000 7f21a0ff9700 1 -- 172.21.15.3:0/3836340955 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f219401bcb0 con 0x7f21a4072520 2023-11-15T05:53:51.046 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.046 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:51.046 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "412cc4c0-837b-11ee-8dbe-212e2dc638e7", 2023-11-15T05:53:51.046 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-15T05:53:51.047 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T05:53:51.047 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T05:53:51.047 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T05:53:51.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T05:53:51.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T05:53:51.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-15T05:53:51.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:51.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T05:53:51.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-15T05:53:51.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:51.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-15T05:53:51.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T05:53:51.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:51.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T05:53:51.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T05:53:51.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T05:53:51.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:51.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T05:53:51.053 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T05:53:51.053 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T05:53:51.053 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T05:53:51.054 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T05:53:51.054 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T05:53:51.054 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T05:53:51.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T05:53:51.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T05:53:51.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T05:53:51.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T05:53:51.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T05:53:51.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T05:53:51.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T05:53:51.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T05:53:51.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.058 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T05:53:51.058 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:51.058 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T05:53:51.058 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T05:53:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-15T05:53:51.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-15T05:53:51.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-15T05:53:51.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:51.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T05:53:51.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T05:53:51.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-15T05:53:37.088356+0000", 2023-11-15T05:53:51.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-15T05:53:51.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-15T05:53:51.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 msgr2=0x7f2190032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 0x7f2190032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f21980096a0 tx=0x7f2198006b40).stop 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 msgr2=0x7f21a410cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f219400e040 tx=0x7f2194006ee0).stop 2023-11-15T05:53:51.062 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 shutdown_connections 2023-11-15T05:53:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f219002fd50 0x7f2190032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 --2- 172.21.15.3:0/3836340955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21a4072520 0x7f21a410cd30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.035+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 >> 172.21.15.3:0/3836340955 conn(0x7f21a40f74b0 msgr2=0x7f21a40f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.036+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 shutdown_connections 2023-11-15T05:53:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.036+0000 7f21aa876700 1 -- 172.21.15.3:0/3836340955 wait complete. 2023-11-15T05:53:51.390 INFO:teuthology.orchestra.run.smithi003.stderr:mgr is available 2023-11-15T05:53:51.390 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling cephadm module... 2023-11-15T05:53:51.391 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T05:53:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.757+0000 7ff5ec053700 1 Processor -- start 2023-11-15T05:53:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.757+0000 7ff5ec053700 1 -- start start 2023-11-15T05:53:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.757+0000 7ff5ec053700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.757+0000 7ff5ec053700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40fb490 con 0x7ff5e40fab80 2023-11-15T05:53:51.759 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e9def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:51.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e9def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34888/0 (socket says 172.21.15.3:34888) 2023-11-15T05:53:51.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e9def700 1 -- 172.21.15.3:0/1010968254 learned_addr learned my addr 172.21.15.3:0/1010968254 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:51.763 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e9def700 1 -- 172.21.15.3:0/1010968254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40fb5d0 con 0x7ff5e40fab80 2023-11-15T05:53:51.763 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e9def700 1 --2- 172.21.15.3:0/1010968254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff5e000b5f0 tx=0x7ff5e0006ee0).ready entity=mon.0 client_cookie=2431840d4ce37b2e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:51.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e8ded700 1 -- 172.21.15.3:0/1010968254 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e000d600 con 0x7ff5e40fab80 2023-11-15T05:53:51.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e8ded700 1 -- 172.21.15.3:0/1010968254 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff5e000dbd0 con 0x7ff5e40fab80 2023-11-15T05:53:51.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.758+0000 7ff5e8ded700 1 -- 172.21.15.3:0/1010968254 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e0011c00 con 0x7ff5e40fab80 2023-11-15T05:53:51.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 -- 172.21.15.3:0/1010968254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 msgr2=0x7ff5e40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:51.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1010968254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7ff5e000b5f0 tx=0x7ff5e0006ee0).stop 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 -- 172.21.15.3:0/1010968254 shutdown_connections 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1010968254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e40fafa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 -- 172.21.15.3:0/1010968254 >> 172.21.15.3:0/1010968254 conn(0x7ff5e40f63c0 msgr2=0x7ff5e40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 -- 172.21.15.3:0/1010968254 shutdown_connections 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.759+0000 7ff5ec053700 1 -- 172.21.15.3:0/1010968254 wait complete. 2023-11-15T05:53:51.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5ec053700 1 Processor -- start 2023-11-15T05:53:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5ec053700 1 -- start start 2023-11-15T05:53:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5ec053700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5ec053700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40ff290 con 0x7ff5e40fab80 2023-11-15T05:53:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5e9def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5e9def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34900/0 (socket says 172.21.15.3:34900) 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5e9def700 1 -- 172.21.15.3:0/1533616175 learned_addr learned my addr 172.21.15.3:0/1533616175 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.760+0000 7ff5e9def700 1 -- 172.21.15.3:0/1533616175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e000b200 con 0x7ff5e40fab80 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5e9def700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff5e000e040 tx=0x7ff5e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e001b410 con 0x7ff5e40fab80 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff5e001b9e0 con 0x7ff5e40fab80 2023-11-15T05:53:51.767 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e40ff430 con 0x7ff5e40fab80 2023-11-15T05:53:51.768 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e001a820 con 0x7ff5e40fab80 2023-11-15T05:53:51.768 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.761+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e40ff840 con 0x7ff5e40fab80 2023-11-15T05:53:51.768 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.762+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7ff5e000f070 con 0x7ff5e40fab80 2023-11-15T05:53:51.769 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.762+0000 7ff5d27fc700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 0x7ff5cc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:51.769 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.762+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff5e0041260 con 0x7ff5e40fab80 2023-11-15T05:53:51.770 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.762+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4045ad0 con 0x7ff5e40fab80 2023-11-15T05:53:51.770 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.762+0000 7ff5e95ee700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 0x7ff5cc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:51.770 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.763+0000 7ff5e95ee700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 0x7ff5cc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff5d40096a0 tx=0x7ff5d4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:51.770 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.767+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5e0016070 con 0x7ff5e40fab80 2023-11-15T05:53:51.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:51.034569+0000 mon.smithi003 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.3:0/3836340955' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T05:53:51.976 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:51.974+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff5e40fc890 con 0x7ff5e40fab80 2023-11-15T05:53:52.619 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.618+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff5e001bb50 con 0x7ff5e40fab80 2023-11-15T05:53:52.620 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.619+0000 7ff5d27fc700 1 -- 172.21.15.3:0/1533616175 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff5e0023d80 con 0x7ff5e40fab80 2023-11-15T05:53:52.622 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 msgr2=0x7ff5cc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:52.623 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 0x7ff5cc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff5d40096a0 tx=0x7ff5d4006b40).stop 2023-11-15T05:53:52.623 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 msgr2=0x7ff5e4100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:52.624 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff5e000e040 tx=0x7ff5e0006ee0).stop 2023-11-15T05:53:52.624 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 shutdown_connections 2023-11-15T05:53:52.624 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7ff5cc02fd50 0x7ff5cc032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:52.625 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 --2- 172.21.15.3:0/1533616175 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff5e40fab80 0x7ff5e4100e10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:52.625 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.621+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 >> 172.21.15.3:0/1533616175 conn(0x7ff5e40f63c0 msgr2=0x7ff5e4070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:52.625 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.622+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 shutdown_connections 2023-11-15T05:53:52.625 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:52.622+0000 7ff5ec053700 1 -- 172.21.15.3:0/1533616175 wait complete. 2023-11-15T05:53:52.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:51.975889+0000 mon.smithi003 (mon.0) 26 : audit [INF] from='client.? 172.21.15.3:0/1533616175' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T05:53:53.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:52.620049+0000 mon.smithi003 (mon.0) 27 : audit [INF] from='client.? 172.21.15.3:0/1533616175' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T05:53:53.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:52.620143+0000 mon.smithi003 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi003.cldzke(active, since 5s) 2023-11-15T05:53:54.184 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T05:53:55.088 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.086+0000 7f4a2069d700 1 Processor -- start 2023-11-15T05:53:55.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.086+0000 7f4a2069d700 1 -- start start 2023-11-15T05:53:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a2069d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a2069d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a180fc750 con 0x7f4a180703b0 2023-11-15T05:53:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a1e439700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:55.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a1e439700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38826/0 (socket says 172.21.15.3:38826) 2023-11-15T05:53:55.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a1e439700 1 -- 172.21.15.3:0/695442999 learned_addr learned my addr 172.21.15.3:0/695442999 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:55.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a1e439700 1 -- 172.21.15.3:0/695442999 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a18101250 con 0x7f4a180703b0 2023-11-15T05:53:55.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.087+0000 7f4a1e439700 1 --2- 172.21.15.3:0/695442999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4a1400b5f0 tx=0x7f4a14006ee0).ready entity=mon.0 client_cookie=fc43f1f03075d5eb server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:55.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.088+0000 7f4a1d437700 1 -- 172.21.15.3:0/695442999 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1400d600 con 0x7f4a180703b0 2023-11-15T05:53:55.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.088+0000 7f4a1d437700 1 -- 172.21.15.3:0/695442999 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a1400dbd0 con 0x7f4a180703b0 2023-11-15T05:53:55.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.088+0000 7f4a1d437700 1 -- 172.21.15.3:0/695442999 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a14011c00 con 0x7f4a180703b0 2023-11-15T05:53:55.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.088+0000 7f4a2069d700 1 -- 172.21.15.3:0/695442999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 msgr2=0x7f4a18100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:55.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.088+0000 7f4a2069d700 1 --2- 172.21.15.3:0/695442999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4a1400b5f0 tx=0x7f4a14006ee0).stop 2023-11-15T05:53:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.090+0000 7f4a2069d700 1 -- 172.21.15.3:0/695442999 shutdown_connections 2023-11-15T05:53:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.090+0000 7f4a2069d700 1 --2- 172.21.15.3:0/695442999 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a18100d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.090+0000 7f4a2069d700 1 -- 172.21.15.3:0/695442999 >> 172.21.15.3:0/695442999 conn(0x7f4a180f63c0 msgr2=0x7f4a180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.090+0000 7f4a2069d700 1 -- 172.21.15.3:0/695442999 shutdown_connections 2023-11-15T05:53:55.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.090+0000 7f4a2069d700 1 -- 172.21.15.3:0/695442999 wait complete. 2023-11-15T05:53:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a2069d700 1 Processor -- start 2023-11-15T05:53:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a2069d700 1 -- start start 2023-11-15T05:53:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a2069d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a2069d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1810d100 con 0x7f4a180703b0 2023-11-15T05:53:55.100 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a1e439700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a1e439700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38836/0 (socket says 172.21.15.3:38836) 2023-11-15T05:53:55.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.091+0000 7f4a1e439700 1 -- 172.21.15.3:0/1753344233 learned_addr learned my addr 172.21.15.3:0/1753344233 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a1e439700 1 -- 172.21.15.3:0/1753344233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1400b200 con 0x7f4a180703b0 2023-11-15T05:53:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a1e439700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4a1400b190 tx=0x7f4a14006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1401b410 con 0x7f4a180703b0 2023-11-15T05:53:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a1401b9e0 con 0x7f4a180703b0 2023-11-15T05:53:55.102 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1810d300 con 0x7f4a180703b0 2023-11-15T05:53:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1401a880 con 0x7f4a180703b0 2023-11-15T05:53:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.092+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1810d710 con 0x7f4a180703b0 2023-11-15T05:53:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4a1400f070 con 0x7f4a180703b0 2023-11-15T05:53:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a18045ad0 con 0x7f4a180703b0 2023-11-15T05:53:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a06ffd700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 0x7f4a00032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4a14041120 con 0x7f4a180703b0 2023-11-15T05:53:55.104 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a1dc38700 1 -- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 msgr2=0x7f4a00032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.093+0000 7f4a1dc38700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 0x7f4a00032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:53:55.105 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.097+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a14022d20 con 0x7f4a180703b0 2023-11-15T05:53:55.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.294+0000 7f4a1dc38700 1 -- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 msgr2=0x7f4a00032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:55.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.294+0000 7f4a1dc38700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 0x7f4a00032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:53:55.308 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.306+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f4a1810e7c0 con 0x7f4a180703b0 2023-11-15T05:53:55.308 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.307+0000 7f4a06ffd700 1 -- 172.21.15.3:0/1753344233 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f4a14022310 con 0x7f4a180703b0 2023-11-15T05:53:55.308 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:55.308 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T05:53:55.310 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:53:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "active_name": "smithi003.cldzke", 2023-11-15T05:53:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T05:53:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 msgr2=0x7f4a00032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:53:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 0x7f4a00032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:53:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 msgr2=0x7f4a1810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4a1400b190 tx=0x7f4a14006ee0).stop 2023-11-15T05:53:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 shutdown_connections 2023-11-15T05:53:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f4a0002fdb0 0x7f4a00032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:53:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.308+0000 7f4a2069d700 1 --2- 172.21.15.3:0/1753344233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a180703b0 0x7f4a1810cc10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.309+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 >> 172.21.15.3:0/1753344233 conn(0x7f4a180f63c0 msgr2=0x7f4a180f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.309+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 shutdown_connections 2023-11-15T05:53:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:55.309+0000 7f4a2069d700 1 -- 172.21.15.3:0/1753344233 wait complete. 2023-11-15T05:53:55.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:55 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:55.307693+0000 mon.smithi003 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.3:0/1753344233' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T05:53:55.705 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for the mgr to restart... 2023-11-15T05:53:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr epoch 5... 2023-11-15T05:53:55.706 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T05:53:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.059+0000 7f07efe33700 1 Processor -- start 2023-11-15T05:53:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07efe33700 1 -- start start 2023-11-15T05:53:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07efe33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07efe33700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80fedd0 con 0x7f07e8070430 2023-11-15T05:53:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07edbcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07edbcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38850/0 (socket says 172.21.15.3:38850) 2023-11-15T05:53:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.060+0000 7f07edbcf700 1 -- 172.21.15.3:0/641491602 learned_addr learned my addr 172.21.15.3:0/641491602 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.061+0000 7f07edbcf700 1 -- 172.21.15.3:0/641491602 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80fef10 con 0x7f07e8070430 2023-11-15T05:53:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.061+0000 7f07edbcf700 1 --2- 172.21.15.3:0/641491602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f07e400b5f0 tx=0x7f07e4006ee0).ready entity=mon.0 client_cookie=dcdf2ee98d88c0b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.061+0000 7f07ecbcd700 1 -- 172.21.15.3:0/641491602 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e400d600 con 0x7f07e8070430 2023-11-15T05:53:56.069 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.061+0000 7f07ecbcd700 1 -- 172.21.15.3:0/641491602 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f07e400dbd0 con 0x7f07e8070430 2023-11-15T05:53:56.069 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.061+0000 7f07ecbcd700 1 -- 172.21.15.3:0/641491602 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e4011c00 con 0x7f07e8070430 2023-11-15T05:53:56.069 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.062+0000 7f07efe33700 1 -- 172.21.15.3:0/641491602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 msgr2=0x7f07e80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.062+0000 7f07efe33700 1 --2- 172.21.15.3:0/641491602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f07e400b5f0 tx=0x7f07e4006ee0).stop 2023-11-15T05:53:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.062+0000 7f07efe33700 1 -- 172.21.15.3:0/641491602 shutdown_connections 2023-11-15T05:53:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.062+0000 7f07efe33700 1 --2- 172.21.15.3:0/641491602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fe8e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.062+0000 7f07efe33700 1 -- 172.21.15.3:0/641491602 >> 172.21.15.3:0/641491602 conn(0x7f07e80f63c0 msgr2=0x7f07e80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:56.071 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 -- 172.21.15.3:0/641491602 shutdown_connections 2023-11-15T05:53:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 -- 172.21.15.3:0/641491602 wait complete. 2023-11-15T05:53:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 Processor -- start 2023-11-15T05:53:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 -- start start 2023-11-15T05:53:56.073 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:56.073 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.063+0000 7f07efe33700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80fa7e0 con 0x7f07e8070430 2023-11-15T05:53:56.073 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07edbcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:56.074 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07edbcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38856/0 (socket says 172.21.15.3:38856) 2023-11-15T05:53:56.077 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07edbcf700 1 -- 172.21.15.3:0/2225101259 learned_addr learned my addr 172.21.15.3:0/2225101259 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:53:56.078 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07edbcf700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e400b200 con 0x7f07e8070430 2023-11-15T05:53:56.078 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07edbcf700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f07e400e040 tx=0x7f07e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:56.078 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e400d9f0 con 0x7f07e8070430 2023-11-15T05:53:56.078 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e80fa980 con 0x7f07e8070430 2023-11-15T05:53:56.079 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f07e401a9f0 con 0x7f07e8070430 2023-11-15T05:53:56.079 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e80fad90 con 0x7f07e8070430 2023-11-15T05:53:56.079 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.064+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e401b8c0 con 0x7f07e8070430 2023-11-15T05:53:56.079 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.065+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f07e400f070 con 0x7f07e8070430 2023-11-15T05:53:56.079 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.065+0000 7f07d67fc700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:56.080 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.065+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d00328d0 con 0x7f07d002fd50 2023-11-15T05:53:56.080 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.066+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f07e4042040 con 0x7f07e8070430 2023-11-15T05:53:56.080 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.066+0000 7f07ed3ce700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 msgr2=0x7f07d0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:56.080 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.066+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:53:56.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.267+0000 7f07ed3ce700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 msgr2=0x7f07d0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:56.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.267+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:53:56.669 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.668+0000 7f07ed3ce700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 msgr2=0x7f07d0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:56.669 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:56.668+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:53:57.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:57.469+0000 7f07ed3ce700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 msgr2=0x7f07d0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3717219179 2023-11-15T05:53:57.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:57.469+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:53:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:58.308+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f07e401ba20 con 0x7f07e8070430 2023-11-15T05:53:58.309 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:58.308+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 msgr2=0x7f07d0032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:53:58.310 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:58.308+0000 7f07d67fc700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:53:58.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:58.256652+0000 mon.smithi003 (mon.0) 30 : cluster [INF] Active manager daemon smithi003.cldzke restarted 2023-11-15T05:53:58.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:58.257092+0000 mon.smithi003 (mon.0) 31 : cluster [INF] Activating manager daemon smithi003.cldzke 2023-11-15T05:53:58.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:58.259552+0000 mon.smithi003 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T05:53:59.316 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.315+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f07e40224f0 con 0x7f07e8070430 2023-11-15T05:53:59.317 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.315+0000 7f07d67fc700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 0x7f07d0035c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:53:59.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.315+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d00328d0 con 0x7f07d0033870 2023-11-15T05:53:59.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.316+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 0x7f07d0035c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:53:59.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.317+0000 7f07ed3ce700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 0x7f07d0035c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f07d80036d0 tx=0x7f07d8006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:53:59.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.318+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f07d00328d0 con 0x7f07d0033870 2023-11-15T05:53:59.325 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.324+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f07e8059100 con 0x7f07d0033870 2023-11-15T05:53:59.326 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07d67fc700 1 -- 172.21.15.3:0/2225101259 <== mgr.14116 v2:172.21.15.3:6800/3770794718 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f07e8059100 con 0x7f07d0033870 2023-11-15T05:53:59.327 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T05:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T05:53:59.328 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:53:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 msgr2=0x7f07d0035c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 0x7f07d0035c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f07d80036d0 tx=0x7f07d8006ca0).stop 2023-11-15T05:53:59.330 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 msgr2=0x7f07e80fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:53:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f07e400e040 tx=0x7f07e4006ee0).stop 2023-11-15T05:53:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 shutdown_connections 2023-11-15T05:53:59.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f07d0033870 0x7f07d0035c60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:59.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07e8070430 0x7f07e80fc360 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:53:59.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 --2- 172.21.15.3:0/2225101259 >> [v2:172.21.15.3:6800/3717219179,v1:172.21.15.3:6801/3717219179] conn(0x7f07d002fd50 0x7f07d0032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:53:59.333 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 >> 172.21.15.3:0/2225101259 conn(0x7f07e80f63c0 msgr2=0x7f07e80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:53:59.333 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 shutdown_connections 2023-11-15T05:53:59.334 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:53:59.325+0000 7f07efe33700 1 -- 172.21.15.3:0/2225101259 wait complete. 2023-11-15T05:53:59.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:58.309723+0000 mon.smithi003 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi003.cldzke(active, starting, since 0.0527124s) 2023-11-15T05:53:59.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.312545+0000 mon.smithi003 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-15T05:53:59.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.313494+0000 mon.smithi003 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mgr metadata", "who": "smithi003.cldzke", "id": "smithi003.cldzke"}]: dispatch 2023-11-15T05:53:59.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.314380+0000 mon.smithi003 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:53:59.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.314652+0000 mon.smithi003 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:53:59.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.314933+0000 mon.smithi003 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:53:59.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:58.328492+0000 mon.smithi003 (mon.0) 39 : cluster [INF] Manager daemon smithi003.cldzke is now available 2023-11-15T05:53:59.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.344209+0000 mon.smithi003 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.347226+0000 mon.smithi003 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.349986+0000 mon.smithi003 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.454066+0000 mon.smithi003 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.454820+0000 mon.smithi003 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.457551+0000 mon.smithi003 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:53:59.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.462386+0000 mon.smithi003 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:53:59.684 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.469694+0000 mon.smithi003 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:53:59.684 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:53:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:58.471064+0000 mon.smithi003 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:53:59.763 INFO:teuthology.orchestra.run.smithi003.stderr:mgr epoch 5 is available 2023-11-15T05:53:59.764 INFO:teuthology.orchestra.run.smithi003.stderr:Setting orchestrator backend to cephadm... 2023-11-15T05:53:59.764 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T05:54:00.143 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.141+0000 7faf02199700 1 Processor -- start 2023-11-15T05:54:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.141+0000 7faf02199700 1 -- start start 2023-11-15T05:54:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.141+0000 7faf02199700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.141+0000 7faf02199700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc0fec80 con 0x7faefc0703d0 2023-11-15T05:54:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.141+0000 7faf01197700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:00.144 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faf01197700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38928/0 (socket says 172.21.15.3:38928) 2023-11-15T05:54:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faf01197700 1 -- 172.21.15.3:0/4058345102 learned_addr learned my addr 172.21.15.3:0/4058345102 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faf01197700 1 -- 172.21.15.3:0/4058345102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc0fedc0 con 0x7faefc0703d0 2023-11-15T05:54:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faf01197700 1 --2- 172.21.15.3:0/4058345102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7faef800b5f0 tx=0x7faef8006ee0).ready entity=mon.0 client_cookie=2299878690bbe8e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faeebfff700 1 -- 172.21.15.3:0/4058345102 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef800d600 con 0x7faefc0703d0 2023-11-15T05:54:00.145 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faeebfff700 1 -- 172.21.15.3:0/4058345102 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faef800dbd0 con 0x7faefc0703d0 2023-11-15T05:54:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.142+0000 7faeebfff700 1 -- 172.21.15.3:0/4058345102 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef8011c00 con 0x7faefc0703d0 2023-11-15T05:54:00.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.143+0000 7faf02199700 1 -- 172.21.15.3:0/4058345102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 msgr2=0x7faefc0fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:00.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.143+0000 7faf02199700 1 --2- 172.21.15.3:0/4058345102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7faef800b5f0 tx=0x7faef8006ee0).stop 2023-11-15T05:54:00.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.143+0000 7faf02199700 1 -- 172.21.15.3:0/4058345102 shutdown_connections 2023-11-15T05:54:00.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.143+0000 7faf02199700 1 --2- 172.21.15.3:0/4058345102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fe790 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:00.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.143+0000 7faf02199700 1 -- 172.21.15.3:0/4058345102 >> 172.21.15.3:0/4058345102 conn(0x7faefc0f6210 msgr2=0x7faefc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:00.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.144+0000 7faf02199700 1 -- 172.21.15.3:0/4058345102 shutdown_connections 2023-11-15T05:54:00.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.144+0000 7faf02199700 1 -- 172.21.15.3:0/4058345102 wait complete. 2023-11-15T05:54:00.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.144+0000 7faf02199700 1 Processor -- start 2023-11-15T05:54:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.144+0000 7faf02199700 1 -- start start 2023-11-15T05:54:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf02199700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf02199700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc0fa690 con 0x7faefc0703d0 2023-11-15T05:54:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf01197700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf01197700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38944/0 (socket says 172.21.15.3:38944) 2023-11-15T05:54:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf01197700 1 -- 172.21.15.3:0/233064297 learned_addr learned my addr 172.21.15.3:0/233064297 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf01197700 1 -- 172.21.15.3:0/233064297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef800b200 con 0x7faefc0703d0 2023-11-15T05:54:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faf01197700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faef800e040 tx=0x7faef8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.145+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef800d9f0 con 0x7faefc0703d0 2023-11-15T05:54:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.146+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faef801a9f0 con 0x7faefc0703d0 2023-11-15T05:54:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.146+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc0fa830 con 0x7faefc0703d0 2023-11-15T05:54:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.146+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef801b8c0 con 0x7faefc0703d0 2023-11-15T05:54:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.146+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc0fac40 con 0x7faefc0703d0 2023-11-15T05:54:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.147+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7faef800f070 con 0x7faefc0703d0 2023-11-15T05:54:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.147+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc045ad0 con 0x7faefc0703d0 2023-11-15T05:54:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.147+0000 7faee9ffb700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 0x7faee40321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.147+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7faef80411a0 con 0x7faefc0703d0 2023-11-15T05:54:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.147+0000 7faf00996700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 0x7faee40321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:00.156 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.148+0000 7faf00996700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 0x7faee40321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faeec0096a0 tx=0x7faeec006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:00.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.152+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faef8022d20 con 0x7faefc0703d0 2023-11-15T05:54:00.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.318+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faef80257e0 con 0x7faefc0703d0 2023-11-15T05:54:00.354 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.354+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7faefc0ffb60 con 0x7faee402fd10 2023-11-15T05:54:00.360 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.359+0000 7faee9ffb700 1 -- 172.21.15.3:0/233064297 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faefc0ffb60 con 0x7faee402fd10 2023-11-15T05:54:00.362 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.360+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 msgr2=0x7faee40321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:00.362 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.360+0000 7faf02199700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 0x7faee40321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faeec0096a0 tx=0x7faeec006b40).stop 2023-11-15T05:54:00.362 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.360+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 msgr2=0x7faefc0fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:00.362 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faef800e040 tx=0x7faef8006ee0).stop 2023-11-15T05:54:00.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 shutdown_connections 2023-11-15T05:54:00.364 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7faee402fd10 0x7faee40321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:00.364 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 --2- 172.21.15.3:0/233064297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faefc0703d0 0x7faefc0fc210 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:00.364 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 >> 172.21.15.3:0/233064297 conn(0x7faefc0f6210 msgr2=0x7faefc0f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:00.364 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 shutdown_connections 2023-11-15T05:54:00.364 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:00.361+0000 7faf02199700 1 -- 172.21.15.3:0/233064297 wait complete. 2023-11-15T05:54:00.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:59.318428+0000 mgr.smithi003.cldzke (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T05:54:00.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:53:59.318827+0000 mon.smithi003 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi003.cldzke(active, since 1.06181s) 2023-11-15T05:54:00.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:53:59.325693+0000 mgr.smithi003.cldzke (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T05:54:01.672 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:00.320472+0000 mon.smithi003 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi003.cldzke(active, since 2s) 2023-11-15T05:54:01.672 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:00.355396+0000 mgr.smithi003.cldzke (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:01.672 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:00.359446+0000 mon.smithi003 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:01.672 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:00.363140+0000 mon.smithi003 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:02.173 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T05:54:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.333+0000 7f281a46c700 1 Processor -- start 2023-11-15T05:54:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f281a46c700 1 -- start start 2023-11-15T05:54:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f281a46c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f281a46c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140fee30 con 0x7f28140703d0 2023-11-15T05:54:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f2813fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f2813fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38956/0 (socket says 172.21.15.3:38956) 2023-11-15T05:54:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f2813fff700 1 -- 172.21.15.3:0/3311188503 learned_addr learned my addr 172.21.15.3:0/3311188503 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f2813fff700 1 -- 172.21.15.3:0/3311188503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28140fef70 con 0x7f28140703d0 2023-11-15T05:54:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.334+0000 7f2813fff700 1 --2- 172.21.15.3:0/3311188503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f280400b5f0 tx=0x7f2804006ee0).ready entity=mon.0 client_cookie=6d51c341f08eef75 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f2812ffd700 1 -- 172.21.15.3:0/3311188503 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f280400d600 con 0x7f28140703d0 2023-11-15T05:54:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f2812ffd700 1 -- 172.21.15.3:0/3311188503 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f280400dbd0 con 0x7f28140703d0 2023-11-15T05:54:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f2812ffd700 1 -- 172.21.15.3:0/3311188503 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2804011c00 con 0x7f28140703d0 2023-11-15T05:54:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 -- 172.21.15.3:0/3311188503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 msgr2=0x7f28140fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 --2- 172.21.15.3:0/3311188503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f280400b5f0 tx=0x7f2804006ee0).stop 2023-11-15T05:54:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 -- 172.21.15.3:0/3311188503 shutdown_connections 2023-11-15T05:54:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 --2- 172.21.15.3:0/3311188503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fe940 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 -- 172.21.15.3:0/3311188503 >> 172.21.15.3:0/3311188503 conn(0x7f28140f63c0 msgr2=0x7f28140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 -- 172.21.15.3:0/3311188503 shutdown_connections 2023-11-15T05:54:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.335+0000 7f281a46c700 1 -- 172.21.15.3:0/3311188503 wait complete. 2023-11-15T05:54:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f281a46c700 1 Processor -- start 2023-11-15T05:54:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f281a46c700 1 -- start start 2023-11-15T05:54:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f281a46c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f281a46c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140fa840 con 0x7f28140703d0 2023-11-15T05:54:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f2813fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f2813fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38964/0 (socket says 172.21.15.3:38964) 2023-11-15T05:54:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.336+0000 7f2813fff700 1 -- 172.21.15.3:0/3079076199 learned_addr learned my addr 172.21.15.3:0/3079076199 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f2813fff700 1 -- 172.21.15.3:0/3079076199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280400b200 con 0x7f28140703d0 2023-11-15T05:54:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f2813fff700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f280400e040 tx=0x7f2804006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f280400d9f0 con 0x7f28140703d0 2023-11-15T05:54:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f280401a9f0 con 0x7f28140703d0 2023-11-15T05:54:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28140fa9e0 con 0x7f28140703d0 2023-11-15T05:54:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f280401b8c0 con 0x7f28140703d0 2023-11-15T05:54:03.547 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.337+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28140fadf0 con 0x7f28140703d0 2023-11-15T05:54:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.338+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f280400f070 con 0x7f28140703d0 2023-11-15T05:54:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.338+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2814045ad0 con 0x7f28140703d0 2023-11-15T05:54:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.338+0000 7f2810ff9700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 0x7f2800032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.338+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2804041320 con 0x7f28140703d0 2023-11-15T05:54:03.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.338+0000 7f28137fe700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 0x7f2800032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.339+0000 7f28137fe700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 0x7f2800032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28080096a0 tx=0x7f2808006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.343+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2804006a70 con 0x7f28140703d0 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.512+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f28140ffd10 con 0x7f280002fd50 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.512+0000 7f2810ff9700 1 -- 172.21.15.3:0/3079076199 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f28140ffd10 con 0x7f280002fd50 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 msgr2=0x7f2800032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 0x7f2800032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28080096a0 tx=0x7f2808006b40).stop 2023-11-15T05:54:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 msgr2=0x7f28140fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f280400e040 tx=0x7f2804006ee0).stop 2023-11-15T05:54:03.550 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 shutdown_connections 2023-11-15T05:54:03.551 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f280002fd50 0x7f2800032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:03.551 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 --2- 172.21.15.3:0/3079076199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28140703d0 0x7f28140fc3c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:03.551 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 >> 172.21.15.3:0/3079076199 conn(0x7f28140f63c0 msgr2=0x7f28140f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:03.551 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 shutdown_connections 2023-11-15T05:54:03.551 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:03.514+0000 7f281a46c700 1 -- 172.21.15.3:0/3079076199 wait complete. 2023-11-15T05:54:03.552 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: value unchanged 2023-11-15T05:54:03.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:03.513411+0000 mgr.smithi003.cldzke (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:05.580 INFO:teuthology.orchestra.run.smithi003.stderr:Generating ssh key... 2023-11-15T05:54:05.580 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T05:54:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2307cb1700 1 Processor -- start 2023-11-15T05:54:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2307cb1700 1 -- start start 2023-11-15T05:54:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2307cb1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2307cb1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000fb4a0 con 0x7f23000fab90 2023-11-15T05:54:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2305a4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2305a4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41772/0 (socket says 172.21.15.3:41772) 2023-11-15T05:54:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.929+0000 7f2305a4d700 1 -- 172.21.15.3:0/3961447901 learned_addr learned my addr 172.21.15.3:0/3961447901 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:07.107 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2305a4d700 1 -- 172.21.15.3:0/3961447901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000fb5e0 con 0x7f23000fab90 2023-11-15T05:54:07.107 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2305a4d700 1 --2- 172.21.15.3:0/3961447901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f22fc00b5f0 tx=0x7f22fc006ee0).ready entity=mon.0 client_cookie=4c58afaf738e41a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:07.107 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2304a4b700 1 -- 172.21.15.3:0/3961447901 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22fc00d600 con 0x7f23000fab90 2023-11-15T05:54:07.107 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2304a4b700 1 -- 172.21.15.3:0/3961447901 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f22fc00dbd0 con 0x7f23000fab90 2023-11-15T05:54:07.107 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2304a4b700 1 -- 172.21.15.3:0/3961447901 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22fc011c00 con 0x7f23000fab90 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2307cb1700 1 -- 172.21.15.3:0/3961447901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 msgr2=0x7f23000fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2307cb1700 1 --2- 172.21.15.3:0/3961447901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f22fc00b5f0 tx=0x7f22fc006ee0).stop 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2307cb1700 1 -- 172.21.15.3:0/3961447901 shutdown_connections 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2307cb1700 1 --2- 172.21.15.3:0/3961447901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f23000fafb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.930+0000 7f2307cb1700 1 -- 172.21.15.3:0/3961447901 >> 172.21.15.3:0/3961447901 conn(0x7f23000f63b0 msgr2=0x7f23000f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:07.108 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 -- 172.21.15.3:0/3961447901 shutdown_connections 2023-11-15T05:54:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 -- 172.21.15.3:0/3961447901 wait complete. 2023-11-15T05:54:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 Processor -- start 2023-11-15T05:54:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 -- start start 2023-11-15T05:54:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2307cb1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230010d0b0 con 0x7f23000fab90 2023-11-15T05:54:07.110 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2305a4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:07.110 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2305a4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41780/0 (socket says 172.21.15.3:41780) 2023-11-15T05:54:07.110 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.931+0000 7f2305a4d700 1 -- 172.21.15.3:0/2753685101 learned_addr learned my addr 172.21.15.3:0/2753685101 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:07.110 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f2305a4d700 1 -- 172.21.15.3:0/2753685101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc00b200 con 0x7f23000fab90 2023-11-15T05:54:07.110 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f2305a4d700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f22fc00b1d0 tx=0x7f22fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:07.111 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22fc01b410 con 0x7f23000fab90 2023-11-15T05:54:07.111 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f22fc01b9e0 con 0x7f23000fab90 2023-11-15T05:54:07.111 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230010d2b0 con 0x7f23000fab90 2023-11-15T05:54:07.111 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22fc01a840 con 0x7f23000fab90 2023-11-15T05:54:07.112 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.932+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230010d6c0 con 0x7f23000fab90 2023-11-15T05:54:07.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.933+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f22fc00f070 con 0x7f23000fab90 2023-11-15T05:54:07.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.933+0000 7f22ee7fc700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 0x7f22e8032250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:07.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.933+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f22fc0411f0 con 0x7f23000fab90 2023-11-15T05:54:07.114 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.933+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2300045ad0 con 0x7f23000fab90 2023-11-15T05:54:07.114 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.933+0000 7f230524c700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 0x7f22e8032250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:07.114 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.934+0000 7f230524c700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 0x7f22e8032250 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f22f00096a0 tx=0x7f22f0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:07.115 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:06.938+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22fc016070 con 0x7f23000fab90 2023-11-15T05:54:07.115 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.105+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f230010dfa0 con 0x7f22e802fd90 2023-11-15T05:54:07.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.918+0000 7f22ee7fc700 1 -- 172.21.15.3:0/2753685101 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f230010dfa0 con 0x7f22e802fd90 2023-11-15T05:54:07.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 msgr2=0x7f22e8032250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:07.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 0x7f22e8032250 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f22f00096a0 tx=0x7f22f0006b40).stop 2023-11-15T05:54:07.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 msgr2=0x7f230010cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:07.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f22fc00b1d0 tx=0x7f22fc006ee0).stop 2023-11-15T05:54:07.924 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 shutdown_connections 2023-11-15T05:54:07.924 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f22e802fd90 0x7f22e8032250 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:07.924 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 --2- 172.21.15.3:0/2753685101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f23000fab90 0x7f230010cbc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:07.925 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 >> 172.21.15.3:0/2753685101 conn(0x7f23000f63b0 msgr2=0x7f23000ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:07.925 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 shutdown_connections 2023-11-15T05:54:07.925 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:07.920+0000 7f2307cb1700 1 -- 172.21.15.3:0/2753685101 wait complete. 2023-11-15T05:54:08.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:07 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:07.106376+0000 mgr.smithi003.cldzke (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-15T05:54:08.904 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T05:54:09.042 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:07.915844+0000 mon.smithi003 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:09.043 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:07.917796+0000 mon.smithi003 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:09.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2af14c700 1 Processor -- start 2023-11-15T05:54:09.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2af14c700 1 -- start start 2023-11-15T05:54:09.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2af14c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:09.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2af14c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80fdc50 con 0x7fc2a80fd340 2023-11-15T05:54:09.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2acee8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:09.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2acee8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41792/0 (socket says 172.21.15.3:41792) 2023-11-15T05:54:09.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.287+0000 7fc2acee8700 1 -- 172.21.15.3:0/1286825632 learned_addr learned my addr 172.21.15.3:0/1286825632 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:09.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.288+0000 7fc2acee8700 1 -- 172.21.15.3:0/1286825632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80fdd90 con 0x7fc2a80fd340 2023-11-15T05:54:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.288+0000 7fc2acee8700 1 --2- 172.21.15.3:0/1286825632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc29c00b5f0 tx=0x7fc29c006ee0).ready entity=mon.0 client_cookie=1303531132780b11 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:09.292 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.288+0000 7fc2a77fe700 1 -- 172.21.15.3:0/1286825632 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29c00d600 con 0x7fc2a80fd340 2023-11-15T05:54:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.288+0000 7fc2a77fe700 1 -- 172.21.15.3:0/1286825632 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc29c00dbd0 con 0x7fc2a80fd340 2023-11-15T05:54:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.288+0000 7fc2a77fe700 1 -- 172.21.15.3:0/1286825632 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29c011c00 con 0x7fc2a80fd340 2023-11-15T05:54:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- 172.21.15.3:0/1286825632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 msgr2=0x7fc2a80fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 --2- 172.21.15.3:0/1286825632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc29c00b5f0 tx=0x7fc29c006ee0).stop 2023-11-15T05:54:09.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- 172.21.15.3:0/1286825632 shutdown_connections 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 --2- 172.21.15.3:0/1286825632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fd760 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- 172.21.15.3:0/1286825632 >> 172.21.15.3:0/1286825632 conn(0x7fc2a80f5b40 msgr2=0x7fc2a80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- 172.21.15.3:0/1286825632 shutdown_connections 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- 172.21.15.3:0/1286825632 wait complete. 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 Processor -- start 2023-11-15T05:54:09.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.289+0000 7fc2af14c700 1 -- start start 2023-11-15T05:54:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2af14c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2af14c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80fb090 con 0x7fc2a80fd340 2023-11-15T05:54:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2acee8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2acee8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41808/0 (socket says 172.21.15.3:41808) 2023-11-15T05:54:09.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2acee8700 1 -- 172.21.15.3:0/2148128201 learned_addr learned my addr 172.21.15.3:0/2148128201 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2acee8700 1 -- 172.21.15.3:0/2148128201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c00b200 con 0x7fc2a80fd340 2023-11-15T05:54:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2acee8700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc29c002f40 tx=0x7fc29c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29c01b410 con 0x7fc2a80fd340 2023-11-15T05:54:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc29c01b9e0 con 0x7fc2a80fd340 2023-11-15T05:54:09.299 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.290+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a80fb230 con 0x7fc2a80fd340 2023-11-15T05:54:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.291+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29c01a8b0 con 0x7fc2a80fd340 2023-11-15T05:54:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.291+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a80fb640 con 0x7fc2a80fd340 2023-11-15T05:54:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.291+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc29c00f070 con 0x7fc2a80fd340 2023-11-15T05:54:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.292+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a8045ad0 con 0x7fc2a80fd340 2023-11-15T05:54:09.300 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.292+0000 7fc2a57fa700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 0x7fc28c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.292+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc29c041120 con 0x7fc2a80fd340 2023-11-15T05:54:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.292+0000 7fc2a7fff700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 0x7fc28c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.292+0000 7fc2a7fff700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 0x7fc28c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc2940096a0 tx=0x7fc294006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:09.301 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.296+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc29c022d20 con 0x7fc2a80fd340 2023-11-15T05:54:09.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.464+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2a80fb880 con 0x7fc28c02fdb0 2023-11-15T05:54:09.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.465+0000 7fc2a57fa700 1 -- 172.21.15.3:0/2148128201 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc2a80fb880 con 0x7fc28c02fdb0 2023-11-15T05:54:09.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxPBVAYPM6VLE1GY1E3IaalzbSwAvuTKN8Rh93YagBfBwfrdkhW3rTAu8BaoHUE0f9pbp6NksrB6miyKZld9+iGvRdZ5mLHR1ZHMDddZRfT2AoWqtOAoxqvyS6r5WbwRUtOC8PmZznNjpqZ+3Wk2dIeFTGomRsLIzyvmeBdHmRnyMX+xGWm3J5G0kA3ZeZoltsnySIy4e7YszPdZ2N+WNU/n/ZxVB0q7qHuSGo9t4oIhJJSblbL3pitRJAmwG9IjUzr1byZziakqlGqYtRJCqD9BSBEsY5l+wUTyF4llgaOX+qd9hVl7rMjil8OCKgDiCqPZ9Wzw5QdrWOIY7ZcsjukQEG7+vLd7mIdBRugKGcbYAVmYcj1HVz59IBVjtHlRD7QrLCAPH7QbXn3UrB+E3BV/wbUThkIr8g9F7RoIZ/zdX+2sxr6QpYXhEFG7A0LEykA7eEz4i/kPMPRdmwAz/kLRyg0rWsbh8rqCJgUfyJybXot5BOtSDbJqicT93uB3U= ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:54:09.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.466+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 msgr2=0x7fc28c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:09.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.466+0000 7fc2af14c700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 0x7fc28c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc2940096a0 tx=0x7fc294006b40).stop 2023-11-15T05:54:09.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.466+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 msgr2=0x7fc2a80fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.466+0000 7fc2af14c700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc29c002f40 tx=0x7fc29c006ee0).stop 2023-11-15T05:54:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 shutdown_connections 2023-11-15T05:54:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc28c02fdb0 0x7fc28c032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 --2- 172.21.15.3:0/2148128201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc2a80fd340 0x7fc2a80fcc10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 >> 172.21.15.3:0/2148128201 conn(0x7fc2a80f5b40 msgr2=0x7fc2a80ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:09.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 shutdown_connections 2023-11-15T05:54:09.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:09.467+0000 7fc2af14c700 1 -- 172.21.15.3:0/2148128201 wait complete. 2023-11-15T05:54:09.904 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T05:54:09.905 INFO:teuthology.orchestra.run.smithi003.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T05:54:09.906 INFO:teuthology.orchestra.run.smithi003.stderr:Adding host smithi003... 2023-11-15T05:54:09.906 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi003 172.21.15.3 2023-11-15T05:54:10.018 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:09.465710+0000 mgr.smithi003.cldzke (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:10.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.308+0000 7f038f8f5700 1 Processor -- start 2023-11-15T05:54:10.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038f8f5700 1 -- start start 2023-11-15T05:54:10.311 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038f8f5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:10.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038f8f5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880fb490 con 0x7f03880fab80 2023-11-15T05:54:10.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038d691700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:10.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038d691700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41824/0 (socket says 172.21.15.3:41824) 2023-11-15T05:54:10.312 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038d691700 1 -- 172.21.15.3:0/396870081 learned_addr learned my addr 172.21.15.3:0/396870081 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:10.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.309+0000 7f038d691700 1 -- 172.21.15.3:0/396870081 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880fb5d0 con 0x7f03880fab80 2023-11-15T05:54:10.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038d691700 1 --2- 172.21.15.3:0/396870081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f038400b870 tx=0x7f0384006f70).ready entity=mon.0 client_cookie=483cea7d8d9b5179 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:10.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f0377fff700 1 -- 172.21.15.3:0/396870081 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f038400da20 con 0x7f03880fab80 2023-11-15T05:54:10.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f0377fff700 1 -- 172.21.15.3:0/396870081 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0384010470 con 0x7f03880fab80 2023-11-15T05:54:10.313 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f0377fff700 1 -- 172.21.15.3:0/396870081 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f038400f5b0 con 0x7f03880fab80 2023-11-15T05:54:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038f8f5700 1 -- 172.21.15.3:0/396870081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 msgr2=0x7f03880fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038f8f5700 1 --2- 172.21.15.3:0/396870081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f038400b870 tx=0x7f0384006f70).stop 2023-11-15T05:54:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038f8f5700 1 -- 172.21.15.3:0/396870081 shutdown_connections 2023-11-15T05:54:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038f8f5700 1 --2- 172.21.15.3:0/396870081 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f03880fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:10.318 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.310+0000 7f038f8f5700 1 -- 172.21.15.3:0/396870081 >> 172.21.15.3:0/396870081 conn(0x7f03880f63c0 msgr2=0x7f03880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 -- 172.21.15.3:0/396870081 shutdown_connections 2023-11-15T05:54:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 -- 172.21.15.3:0/396870081 wait complete. 2023-11-15T05:54:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 Processor -- start 2023-11-15T05:54:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 -- start start 2023-11-15T05:54:10.319 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.311+0000 7f038f8f5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038810d080 con 0x7f03880fab80 2023-11-15T05:54:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038d691700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038d691700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41838/0 (socket says 172.21.15.3:41838) 2023-11-15T05:54:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038d691700 1 -- 172.21.15.3:0/336635179 learned_addr learned my addr 172.21.15.3:0/336635179 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:10.320 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038d691700 1 -- 172.21.15.3:0/336635179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038400b480 con 0x7f03880fab80 2023-11-15T05:54:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038d691700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0384006a40 tx=0x7f0384006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f038400de00 con 0x7f03880fab80 2023-11-15T05:54:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f038400fef0 con 0x7f03880fab80 2023-11-15T05:54:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038810d280 con 0x7f03880fab80 2023-11-15T05:54:10.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.312+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0384019e90 con 0x7f03880fab80 2023-11-15T05:54:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.313+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038810d690 con 0x7f03880fab80 2023-11-15T05:54:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.314+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0388045ad0 con 0x7f03880fab80 2023-11-15T05:54:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.314+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f038402a070 con 0x7f03880fab80 2023-11-15T05:54:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.314+0000 7f0375ffb700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 0x7f0370032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:10.322 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.314+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0384041b20 con 0x7f03880fab80 2023-11-15T05:54:10.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.314+0000 7f038ce90700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 0x7f0370032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:10.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.315+0000 7f038ce90700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 0x7f0370032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f03780096a0 tx=0x7f0378006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:10.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.318+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0384019630 con 0x7f03880fab80 2023-11-15T05:54:10.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:10.509+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}) v1 -- 0x7f038810df70 con 0x7f037002fdb0 2023-11-15T05:54:11.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Added host 'smithi003' 2023-11-15T05:54:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.156+0000 7f0375ffb700 1 -- 172.21.15.3:0/336635179 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f038810df70 con 0x7f037002fdb0 2023-11-15T05:54:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 msgr2=0x7f0370032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:11.160 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 0x7f0370032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f03780096a0 tx=0x7f0378006b40).stop 2023-11-15T05:54:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 msgr2=0x7f038810cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0384006a40 tx=0x7f0384006f70).stop 2023-11-15T05:54:11.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 shutdown_connections 2023-11-15T05:54:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f037002fdb0 0x7f0370032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 --2- 172.21.15.3:0/336635179 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03880fab80 0x7f038810cb90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 >> 172.21.15.3:0/336635179 conn(0x7f03880f63c0 msgr2=0x7f03880ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 shutdown_connections 2023-11-15T05:54:11.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.159+0000 7f038f8f5700 1 -- 172.21.15.3:0/336635179 wait complete. 2023-11-15T05:54:11.487 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying mon service with default placement... 2023-11-15T05:54:11.488 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T05:54:11.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.908+0000 7fd33594f700 1 Processor -- start 2023-11-15T05:54:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.908+0000 7fd33594f700 1 -- start start 2023-11-15T05:54:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd33594f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd33594f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300fc5c0 con 0x7fd3300fbcb0 2023-11-15T05:54:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41846/0 (socket says 172.21.15.3:41846) 2023-11-15T05:54:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32effd700 1 -- 172.21.15.3:0/760558797 learned_addr learned my addr 172.21.15.3:0/760558797 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32effd700 1 -- 172.21.15.3:0/760558797 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300fc700 con 0x7fd3300fbcb0 2023-11-15T05:54:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32effd700 1 --2- 172.21.15.3:0/760558797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fd32000b5f0 tx=0x7fd320006ee0).ready entity=mon.0 client_cookie=c9545f085f09794 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.909+0000 7fd32dffb700 1 -- 172.21.15.3:0/760558797 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd32000d600 con 0x7fd3300fbcb0 2023-11-15T05:54:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd32dffb700 1 -- 172.21.15.3:0/760558797 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd32000dbd0 con 0x7fd3300fbcb0 2023-11-15T05:54:11.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd32dffb700 1 -- 172.21.15.3:0/760558797 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd320011c00 con 0x7fd3300fbcb0 2023-11-15T05:54:11.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 -- 172.21.15.3:0/760558797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 msgr2=0x7fd3300fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:11.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 --2- 172.21.15.3:0/760558797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fd32000b5f0 tx=0x7fd320006ee0).stop 2023-11-15T05:54:11.917 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 -- 172.21.15.3:0/760558797 shutdown_connections 2023-11-15T05:54:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 --2- 172.21.15.3:0/760558797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300fc0d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 -- 172.21.15.3:0/760558797 >> 172.21.15.3:0/760558797 conn(0x7fd3300f74b0 msgr2=0x7fd3300f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 -- 172.21.15.3:0/760558797 shutdown_connections 2023-11-15T05:54:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.910+0000 7fd33594f700 1 -- 172.21.15.3:0/760558797 wait complete. 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.911+0000 7fd33594f700 1 Processor -- start 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd33594f700 1 -- start start 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd33594f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd33594f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330070a60 con 0x7fd3300fbcb0 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd32effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd32effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41862/0 (socket says 172.21.15.3:41862) 2023-11-15T05:54:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd32effd700 1 -- 172.21.15.3:0/2313753133 learned_addr learned my addr 172.21.15.3:0/2313753133 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd32effd700 1 -- 172.21.15.3:0/2313753133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32000b200 con 0x7fd3300fbcb0 2023-11-15T05:54:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.912+0000 7fd32effd700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd32000e040 tx=0x7fd320006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.913+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd32001b410 con 0x7fd3300fbcb0 2023-11-15T05:54:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.913+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd32001b9e0 con 0x7fd3300fbcb0 2023-11-15T05:54:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.913+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330070c00 con 0x7fd3300fbcb0 2023-11-15T05:54:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.913+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd32001a820 con 0x7fd3300fbcb0 2023-11-15T05:54:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.913+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd330071010 con 0x7fd3300fbcb0 2023-11-15T05:54:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.914+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330045ad0 con 0x7fd3300fbcb0 2023-11-15T05:54:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.914+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd32000f070 con 0x7fd3300fbcb0 2023-11-15T05:54:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.914+0000 7fd3177fe700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 0x7fd31c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.914+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd320041380 con 0x7fd3300fbcb0 2023-11-15T05:54:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.914+0000 7fd32e7fc700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 0x7fd31c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.915+0000 7fd32e7fc700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 0x7fd31c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3240096a0 tx=0x7fd324006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:11.918+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd320022310 con 0x7fd3300fbcb0 2023-11-15T05:54:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:10.510960+0000 mgr.smithi003.cldzke (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:11.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:11.156359+0000 mon.smithi003 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:11.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:11.156973+0000 mon.smithi003 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:12.113 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.112+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fd330071250 con 0x7fd31c02fd50 2023-11-15T05:54:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.116+0000 7fd3177fe700 1 -- 172.21.15.3:0/2313753133 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd330071250 con 0x7fd31c02fd50 2023-11-15T05:54:12.117 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T05:54:12.120 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 msgr2=0x7fd31c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:12.120 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 0x7fd31c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3240096a0 tx=0x7fd324006b40).stop 2023-11-15T05:54:12.120 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 msgr2=0x7fd3300725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:12.121 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fd32000e040 tx=0x7fd320006ee0).stop 2023-11-15T05:54:12.121 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 shutdown_connections 2023-11-15T05:54:12.121 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fd31c02fd50 0x7fd31c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:12.121 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 --2- 172.21.15.3:0/2313753133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3300fbcb0 0x7fd3300725e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:12.122 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.118+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 >> 172.21.15.3:0/2313753133 conn(0x7fd3300f74b0 msgr2=0x7fd3300f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:12.122 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.119+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 shutdown_connections 2023-11-15T05:54:12.122 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.119+0000 7fd33594f700 1 -- 172.21.15.3:0/2313753133 wait complete. 2023-11-15T05:54:12.420 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying mgr service with default placement... 2023-11-15T05:54:12.420 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T05:54:12.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:11.156711+0000 mgr.smithi003.cldzke (mgr.14116) 9 : cephadm [INF] Added host smithi003 2023-11-15T05:54:12.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:12.116895+0000 mon.smithi003 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:12.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.847+0000 7f6914edb700 1 Processor -- start 2023-11-15T05:54:12.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.847+0000 7f6914edb700 1 -- start start 2023-11-15T05:54:12.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f6914edb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:12.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f6914edb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69100fb490 con 0x7f69100fab80 2023-11-15T05:54:12.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:12.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41866/0 (socket says 172.21.15.3:41866) 2023-11-15T05:54:12.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690e59c700 1 -- 172.21.15.3:0/1810529163 learned_addr learned my addr 172.21.15.3:0/1810529163 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:12.853 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690e59c700 1 -- 172.21.15.3:0/1810529163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100fb5d0 con 0x7f69100fab80 2023-11-15T05:54:12.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690e59c700 1 --2- 172.21.15.3:0/1810529163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f68f800b5f0 tx=0x7f68f8006ee0).ready entity=mon.0 client_cookie=480c8afb3ec1780 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:12.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690d59a700 1 -- 172.21.15.3:0/1810529163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f800d600 con 0x7f69100fab80 2023-11-15T05:54:12.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.848+0000 7f690d59a700 1 -- 172.21.15.3:0/1810529163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f68f800dbd0 con 0x7f69100fab80 2023-11-15T05:54:12.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f690d59a700 1 -- 172.21.15.3:0/1810529163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f8011c00 con 0x7f69100fab80 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 -- 172.21.15.3:0/1810529163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 msgr2=0x7f69100fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 --2- 172.21.15.3:0/1810529163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f68f800b5f0 tx=0x7f68f8006ee0).stop 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 -- 172.21.15.3:0/1810529163 shutdown_connections 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 --2- 172.21.15.3:0/1810529163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f69100fafa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 -- 172.21.15.3:0/1810529163 >> 172.21.15.3:0/1810529163 conn(0x7f69100f63c0 msgr2=0x7f69100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.849+0000 7f6914edb700 1 -- 172.21.15.3:0/1810529163 shutdown_connections 2023-11-15T05:54:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.850+0000 7f6914edb700 1 -- 172.21.15.3:0/1810529163 wait complete. 2023-11-15T05:54:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.850+0000 7f6914edb700 1 Processor -- start 2023-11-15T05:54:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.850+0000 7f6914edb700 1 -- start start 2023-11-15T05:54:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6914edb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6914edb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910070470 con 0x7f69100fab80 2023-11-15T05:54:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f690e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f690e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41876/0 (socket says 172.21.15.3:41876) 2023-11-15T05:54:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f690e59c700 1 -- 172.21.15.3:0/2235333312 learned_addr learned my addr 172.21.15.3:0/2235333312 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f690e59c700 1 -- 172.21.15.3:0/2235333312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f800b200 con 0x7f69100fab80 2023-11-15T05:54:12.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f690e59c700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f68f800e040 tx=0x7f68f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:12.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f801b410 con 0x7f69100fab80 2023-11-15T05:54:12.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f68f801b9e0 con 0x7f69100fab80 2023-11-15T05:54:12.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f801a820 con 0x7f69100fab80 2023-11-15T05:54:12.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.851+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6910070670 con 0x7f69100fab80 2023-11-15T05:54:12.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.852+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691010ddd0 con 0x7f69100fab80 2023-11-15T05:54:12.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.852+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f68f800f070 con 0x7f69100fab80 2023-11-15T05:54:12.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.852+0000 7f6906ffd700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 0x7f68f0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:12.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.852+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f68f8041310 con 0x7f69100fab80 2023-11-15T05:54:12.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.853+0000 7f690dd9b700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 0x7f68f0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:12.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.853+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6910045ad0 con 0x7f69100fab80 2023-11-15T05:54:12.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.853+0000 7f690dd9b700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 0x7f68f0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f69000096a0 tx=0x7f6900006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:12.861 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:12.857+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68f8016070 con 0x7f69100fab80 2023-11-15T05:54:13.053 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.051+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f69100fc900 con 0x7f68f002fdb0 2023-11-15T05:54:13.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.056+0000 7f6906ffd700 1 -- 172.21.15.3:0/2235333312 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f69100fc900 con 0x7f68f002fdb0 2023-11-15T05:54:13.057 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T05:54:13.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.057+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 msgr2=0x7f68f0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:13.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.057+0000 7f6914edb700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 0x7f68f0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f69000096a0 tx=0x7f6900006b40).stop 2023-11-15T05:54:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.057+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 msgr2=0x7f691010d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.057+0000 7f6914edb700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f68f800e040 tx=0x7f68f8006ee0).stop 2023-11-15T05:54:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.057+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 shutdown_connections 2023-11-15T05:54:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.058+0000 7f6914edb700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f68f002fdb0 0x7f68f0032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.058+0000 7f6914edb700 1 --2- 172.21.15.3:0/2235333312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69100fab80 0x7f691010d720 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:13.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.058+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 >> 172.21.15.3:0/2235333312 conn(0x7f69100f63c0 msgr2=0x7f69100f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:13.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.058+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 shutdown_connections 2023-11-15T05:54:13.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.058+0000 7f6914edb700 1 -- 172.21.15.3:0/2235333312 wait complete. 2023-11-15T05:54:13.420 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying crash service with default placement... 2023-11-15T05:54:13.421 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T05:54:13.627 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:12.113496+0000 mgr.smithi003.cldzke (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:13.627 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:12.114354+0000 mgr.smithi003.cldzke (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T05:54:13.627 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:13.056338+0000 mon.smithi003 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:13.849 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.847+0000 7f4df2b86700 1 Processor -- start 2023-11-15T05:54:13.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.847+0000 7f4df2b86700 1 -- start start 2023-11-15T05:54:13.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df2b86700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:13.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df2b86700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec0fedd0 con 0x7f4dec070430 2023-11-15T05:54:13.850 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df0922700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:13.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df0922700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41892/0 (socket says 172.21.15.3:41892) 2023-11-15T05:54:13.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df0922700 1 -- 172.21.15.3:0/3488233850 learned_addr learned my addr 172.21.15.3:0/3488233850 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:13.851 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df0922700 1 -- 172.21.15.3:0/3488233850 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec0fef10 con 0x7f4dec070430 2023-11-15T05:54:13.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4df0922700 1 --2- 172.21.15.3:0/3488233850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4de000b5f0 tx=0x7f4de0006ee0).ready entity=mon.0 client_cookie=23bf3eceb26c7b28 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:13.854 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4deb7fe700 1 -- 172.21.15.3:0/3488233850 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de000d600 con 0x7f4dec070430 2023-11-15T05:54:13.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4deb7fe700 1 -- 172.21.15.3:0/3488233850 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4de000dbd0 con 0x7f4dec070430 2023-11-15T05:54:13.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.848+0000 7f4deb7fe700 1 -- 172.21.15.3:0/3488233850 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de0011c00 con 0x7f4dec070430 2023-11-15T05:54:13.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 -- 172.21.15.3:0/3488233850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 msgr2=0x7f4dec0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:13.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 --2- 172.21.15.3:0/3488233850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4de000b5f0 tx=0x7f4de0006ee0).stop 2023-11-15T05:54:13.855 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 -- 172.21.15.3:0/3488233850 shutdown_connections 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 --2- 172.21.15.3:0/3488233850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 -- 172.21.15.3:0/3488233850 >> 172.21.15.3:0/3488233850 conn(0x7f4dec0f63c0 msgr2=0x7f4dec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 -- 172.21.15.3:0/3488233850 shutdown_connections 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.849+0000 7f4df2b86700 1 -- 172.21.15.3:0/3488233850 wait complete. 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.850+0000 7f4df2b86700 1 Processor -- start 2023-11-15T05:54:13.856 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.850+0000 7f4df2b86700 1 -- start start 2023-11-15T05:54:13.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.850+0000 7f4df2b86700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:13.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.850+0000 7f4df2b86700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec0fa7e0 con 0x7f4dec070430 2023-11-15T05:54:13.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df0922700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:13.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df0922700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41906/0 (socket says 172.21.15.3:41906) 2023-11-15T05:54:13.857 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df0922700 1 -- 172.21.15.3:0/163270247 learned_addr learned my addr 172.21.15.3:0/163270247 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:13.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df0922700 1 -- 172.21.15.3:0/163270247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de000b200 con 0x7f4dec070430 2023-11-15T05:54:13.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df0922700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4de000e040 tx=0x7f4de0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:13.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de000d9f0 con 0x7f4dec070430 2023-11-15T05:54:13.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec0fa980 con 0x7f4dec070430 2023-11-15T05:54:13.858 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4de001a9f0 con 0x7f4dec070430 2023-11-15T05:54:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de001b8c0 con 0x7f4dec070430 2023-11-15T05:54:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.851+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec0fad90 con 0x7f4dec070430 2023-11-15T05:54:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.852+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4de000f070 con 0x7f4dec070430 2023-11-15T05:54:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.852+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec045ad0 con 0x7f4dec070430 2023-11-15T05:54:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.852+0000 7f4de97fa700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 0x7f4dd0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:13.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.853+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4de0041320 con 0x7f4dec070430 2023-11-15T05:54:13.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.853+0000 7f4debfff700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 0x7f4dd0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:13.860 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.853+0000 7f4debfff700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 0x7f4dd0032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4dd80096a0 tx=0x7f4dd8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:13.861 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:13.857+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4de0022d70 con 0x7f4dec070430 2023-11-15T05:54:14.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.050+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dec0ffd10 con 0x7f4dd002fd50 2023-11-15T05:54:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.055+0000 7f4de97fa700 1 -- 172.21.15.3:0/163270247 <== mgr.14116 v2:172.21.15.3:6800/3770794718 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4dec0ffd10 con 0x7f4dd002fd50 2023-11-15T05:54:14.056 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T05:54:14.058 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.056+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 msgr2=0x7f4dd0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.056+0000 7f4df2b86700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 0x7f4dd0032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4dd80096a0 tx=0x7f4dd8006b40).stop 2023-11-15T05:54:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.056+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 msgr2=0x7f4dec0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4de000e040 tx=0x7f4de0006ee0).stop 2023-11-15T05:54:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 shutdown_connections 2023-11-15T05:54:14.059 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f4dd002fd50 0x7f4dd0032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 --2- 172.21.15.3:0/163270247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dec070430 0x7f4dec0fc360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 >> 172.21.15.3:0/163270247 conn(0x7f4dec0f63c0 msgr2=0x7f4dec0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 shutdown_connections 2023-11-15T05:54:14.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.057+0000 7f4df2b86700 1 -- 172.21.15.3:0/163270247 wait complete. 2023-11-15T05:54:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling mgr prometheus module... 2023-11-15T05:54:14.416 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T05:54:14.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:13.053226+0000 mgr.smithi003.cldzke (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:14.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:13.053813+0000 mgr.smithi003.cldzke (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T05:54:14.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:14.055485+0000 mon.smithi003 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.3:0/3852119804' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:14.782 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.781+0000 7f2e7f553700 1 Processor -- start 2023-11-15T05:54:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.781+0000 7f2e7f553700 1 -- start start 2023-11-15T05:54:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.781+0000 7f2e7f553700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.781+0000 7f2e7f553700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780fc770 con 0x7f2e78070390 2023-11-15T05:54:14.786 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.781+0000 7f2e7d2ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:14.786 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e7d2ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56248/0 (socket says 172.21.15.3:56248) 2023-11-15T05:54:14.786 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e7d2ef700 1 -- 172.21.15.3:0/508851518 learned_addr learned my addr 172.21.15.3:0/508851518 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:14.786 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e7d2ef700 1 -- 172.21.15.3:0/508851518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78101270 con 0x7f2e78070390 2023-11-15T05:54:14.786 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e7d2ef700 1 --2- 172.21.15.3:0/508851518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2e7400b5f0 tx=0x7f2e74006ee0).ready entity=mon.0 client_cookie=4dda560ebb27a225 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:14.787 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e67fff700 1 -- 172.21.15.3:0/508851518 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e7400d600 con 0x7f2e78070390 2023-11-15T05:54:14.787 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e67fff700 1 -- 172.21.15.3:0/508851518 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e7400dbd0 con 0x7f2e78070390 2023-11-15T05:54:14.787 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.782+0000 7f2e67fff700 1 -- 172.21.15.3:0/508851518 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e74011c00 con 0x7f2e78070390 2023-11-15T05:54:14.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 -- 172.21.15.3:0/508851518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 msgr2=0x7f2e78100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:14.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 --2- 172.21.15.3:0/508851518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f2e7400b5f0 tx=0x7f2e74006ee0).stop 2023-11-15T05:54:14.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 -- 172.21.15.3:0/508851518 shutdown_connections 2023-11-15T05:54:14.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 --2- 172.21.15.3:0/508851518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e78070390 0x7f2e78100d80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 -- 172.21.15.3:0/508851518 >> 172.21.15.3:0/508851518 conn(0x7f2e780f63c0 msgr2=0x7f2e780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 -- 172.21.15.3:0/508851518 shutdown_connections 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.783+0000 7f2e7f553700 1 -- 172.21.15.3:0/508851518 wait complete. 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7f553700 1 Processor -- start 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7f553700 1 -- start start 2023-11-15T05:54:14.789 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7f553700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:14.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7f553700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7810d680 con 0x7f2e7810cd70 2023-11-15T05:54:14.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7d2ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:14.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7d2ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56262/0 (socket says 172.21.15.3:56262) 2023-11-15T05:54:14.790 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.784+0000 7f2e7d2ef700 1 -- 172.21.15.3:0/874828682 learned_addr learned my addr 172.21.15.3:0/874828682 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:14.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e7d2ef700 1 -- 172.21.15.3:0/874828682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7400b200 con 0x7f2e7810cd70 2023-11-15T05:54:14.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e7d2ef700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2e74011820 tx=0x7f2e74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:14.792 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e7401b410 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7810d880 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2e7401b9e0 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e7401a8e0 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.785+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7810e2a0 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.786+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2e7400f070 con 0x7f2e7810cd70 2023-11-15T05:54:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.786+0000 7f2e65ffb700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 0x7f2e60032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.786+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2e74040fc0 con 0x7f2e7810cd70 2023-11-15T05:54:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.787+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e78059100 con 0x7f2e7810cd70 2023-11-15T05:54:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.787+0000 7f2e7caee700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 0x7f2e60032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.787+0000 7f2e7caee700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 0x7f2e60032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2e680096a0 tx=0x7f2e68006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.791+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e74016070 con 0x7f2e7810cd70 2023-11-15T05:54:14.997 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:14.996+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f2e7810ed70 con 0x7f2e7810cd70 2023-11-15T05:54:15.490 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.488+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f2e7401aa40 con 0x7f2e7810cd70 2023-11-15T05:54:15.491 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.490+0000 7f2e65ffb700 1 -- 172.21.15.3:0/874828682 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f2e7402da30 con 0x7f2e7810cd70 2023-11-15T05:54:15.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.492+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 msgr2=0x7f2e60032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:15.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.492+0000 7f2e7f553700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 0x7f2e60032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2e680096a0 tx=0x7f2e68006b40).stop 2023-11-15T05:54:15.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.492+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 msgr2=0x7f2e7810d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:15.494 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.492+0000 7f2e7f553700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2e74011820 tx=0x7f2e74006ee0).stop 2023-11-15T05:54:15.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 shutdown_connections 2023-11-15T05:54:15.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7f2e6002fd50 0x7f2e60032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:15.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 --2- 172.21.15.3:0/874828682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e7810cd70 0x7f2e7810d190 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:15.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 >> 172.21.15.3:0/874828682 conn(0x7f2e780f63c0 msgr2=0x7f2e780f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:15.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 shutdown_connections 2023-11-15T05:54:15.496 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:15.494+0000 7f2e7f553700 1 -- 172.21.15.3:0/874828682 wait complete. 2023-11-15T05:54:15.827 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:14.052304+0000 mgr.smithi003.cldzke (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:15.827 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:14.053011+0000 mgr.smithi003.cldzke (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T05:54:15.827 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:14.997244+0000 mon.smithi003 (mon.0) 60 : audit [INF] from='client.? 172.21.15.3:0/874828682' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T05:54:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying prometheus service with default placement... 2023-11-15T05:54:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T05:54:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.285+0000 7fc089f1b700 1 Processor -- start 2023-11-15T05:54:16.287 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.285+0000 7fc089f1b700 1 -- start start 2023-11-15T05:54:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.285+0000 7fc089f1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.285+0000 7fc089f1b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0840fc8d0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc088f19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc088f19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56286/0 (socket says 172.21.15.3:56286) 2023-11-15T05:54:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc088f19700 1 -- 172.21.15.3:0/3413691380 learned_addr learned my addr 172.21.15.3:0/3413691380 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc088f19700 1 -- 172.21.15.3:0/3413691380 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0840fca10 con 0x7fc0840fbfc0 2023-11-15T05:54:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc088f19700 1 --2- 172.21.15.3:0/3413691380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc07800b5f0 tx=0x7fc078006ee0).ready entity=mon.0 client_cookie=36dc6aa7a64b8823 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:16.289 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc0837fe700 1 -- 172.21.15.3:0/3413691380 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc07800d600 con 0x7fc0840fbfc0 2023-11-15T05:54:16.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc0837fe700 1 -- 172.21.15.3:0/3413691380 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc07800dbd0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.286+0000 7fc0837fe700 1 -- 172.21.15.3:0/3413691380 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc078011c00 con 0x7fc0840fbfc0 2023-11-15T05:54:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 -- 172.21.15.3:0/3413691380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 msgr2=0x7fc0840fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3413691380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc07800b5f0 tx=0x7fc078006ee0).stop 2023-11-15T05:54:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 -- 172.21.15.3:0/3413691380 shutdown_connections 2023-11-15T05:54:16.293 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3413691380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc0840fc3e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:16.294 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 -- 172.21.15.3:0/3413691380 >> 172.21.15.3:0/3413691380 conn(0x7fc0840f59a0 msgr2=0x7fc0840f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:16.294 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 -- 172.21.15.3:0/3413691380 shutdown_connections 2023-11-15T05:54:16.294 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 -- 172.21.15.3:0/3413691380 wait complete. 2023-11-15T05:54:16.294 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.287+0000 7fc089f1b700 1 Processor -- start 2023-11-15T05:54:16.294 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc089f1b700 1 -- start start 2023-11-15T05:54:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc089f1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc089f1b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08410adb0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc088f19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc088f19700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56294/0 (socket says 172.21.15.3:56294) 2023-11-15T05:54:16.295 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc088f19700 1 -- 172.21.15.3:0/3518002735 learned_addr learned my addr 172.21.15.3:0/3518002735 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.288+0000 7fc088f19700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07800b200 con 0x7fc0840fbfc0 2023-11-15T05:54:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc088f19700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc078000c00 tx=0x7fc078006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc078011690 con 0x7fc0840fbfc0 2023-11-15T05:54:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc07800d9f0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08410afb0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc07801b710 con 0x7fc0840fbfc0 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.289+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08410b3c0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc084045ad0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fc07800f070 con 0x7fc0840fbfc0 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc0817fa700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:16.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc0780411e0 con 0x7fc0840fbfc0 2023-11-15T05:54:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc083fff700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3770794718 2023-11-15T05:54:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.290+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:54:16.298 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.295+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc078022900 con 0x7fc0840fbfc0 2023-11-15T05:54:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.491+0000 7fc083fff700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3770794718 2023-11-15T05:54:16.492 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.491+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:54:16.495 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.494+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc08410bd00 con 0x7fc06802fdb0 2023-11-15T05:54:16.894 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.892+0000 7fc083fff700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3770794718 2023-11-15T05:54:16.894 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:16.892+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:54:16.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:15.490772+0000 mon.smithi003 (mon.0) 61 : audit [INF] from='client.? 172.21.15.3:0/874828682' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T05:54:16.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:15.490846+0000 mon.smithi003 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi003.cldzke(active, since 17s) 2023-11-15T05:54:17.694 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:17.693+0000 7fc083fff700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3770794718 2023-11-15T05:54:17.694 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:17.693+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:54:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:19.294+0000 7fc083fff700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/3770794718 2023-11-15T05:54:19.297 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:19.294+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T05:54:21.489 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:21.487+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fc078025460 con 0x7fc0840fbfc0 2023-11-15T05:54:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:21.487+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 msgr2=0x7fc068032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:54:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:21.487+0000 7fc0817fa700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:21.741 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:21.128630+0000 mon.smithi003 (mon.0) 63 : cluster [INF] Active manager daemon smithi003.cldzke restarted 2023-11-15T05:54:21.741 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:21.129075+0000 mon.smithi003 (mon.0) 64 : cluster [INF] Activating manager daemon smithi003.cldzke 2023-11-15T05:54:21.741 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:21.133388+0000 mon.smithi003 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T05:54:22.497 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.494+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fc07803d730 con 0x7fc0840fbfc0 2023-11-15T05:54:22.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.494+0000 7fc0817fa700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 0x7fc068036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:22.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.494+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 --> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc08410bd00 con 0x7fc068033c20 2023-11-15T05:54:22.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.494+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 0x7fc068036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:22.498 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.495+0000 7fc083fff700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 0x7fc068036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc0700036d0 tx=0x7fc070006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:22.502 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.500+0000 7fc0817fa700 1 -- 172.21.15.3:0/3518002735 <== mgr.14144 v2:172.21.15.3:6800/2695659223 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc08410bd00 con 0x7fc068033c20 2023-11-15T05:54:22.502 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T05:54:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.502+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 msgr2=0x7fc068036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.502+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 0x7fc068036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc0700036d0 tx=0x7fc070006cb0).stop 2023-11-15T05:54:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 msgr2=0x7fc08410a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:22.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc078000c00 tx=0x7fc078006ee0).stop 2023-11-15T05:54:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 shutdown_connections 2023-11-15T05:54:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fc068033c20 0x7fc068036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0840fbfc0 0x7fc08410a8c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 --2- 172.21.15.3:0/3518002735 >> [v2:172.21.15.3:6800/3770794718,v1:172.21.15.3:6801/3770794718] conn(0x7fc06802fdb0 0x7fc068032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:22.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 >> 172.21.15.3:0/3518002735 conn(0x7fc0840f59a0 msgr2=0x7fc0840fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:22.508 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 shutdown_connections 2023-11-15T05:54:22.508 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:22.503+0000 7fc089f1b700 1 -- 172.21.15.3:0/3518002735 wait complete. 2023-11-15T05:54:22.860 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying grafana service with default placement... 2023-11-15T05:54:22.860 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T05:54:22.863 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:21.490084+0000 mon.smithi003 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi003.cldzke(active, starting, since 0.361106s) 2023-11-15T05:54:22.863 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.491795+0000 mon.smithi003 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-15T05:54:22.863 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.492540+0000 mon.smithi003 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mgr metadata", "who": "smithi003.cldzke", "id": "smithi003.cldzke"}]: dispatch 2023-11-15T05:54:22.864 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.493509+0000 mon.smithi003 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:54:22.864 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.493728+0000 mon.smithi003 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:54:22.864 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.493897+0000 mon.smithi003 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:54:22.864 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:21.508439+0000 mon.smithi003 (mon.0) 72 : cluster [INF] Manager daemon smithi003.cldzke is now available 2023-11-15T05:54:22.864 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.627298+0000 mon.smithi003 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:22.865 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.628928+0000 mon.smithi003 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:22.865 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.645374+0000 mon.smithi003 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:54:22.865 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:21.648543+0000 mon.smithi003 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:54:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f2465b1e700 1 Processor -- start 2023-11-15T05:54:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f2465b1e700 1 -- start start 2023-11-15T05:54:23.263 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f2465b1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f2465b1e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460070790 con 0x7f2460070370 2023-11-15T05:54:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56360/0 (socket says 172.21.15.3:56360) 2023-11-15T05:54:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f245f7fe700 1 -- 172.21.15.3:0/2137677463 learned_addr learned my addr 172.21.15.3:0/2137677463 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f245f7fe700 1 -- 172.21.15.3:0/2137677463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600fee90 con 0x7f2460070370 2023-11-15T05:54:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.252+0000 7f245f7fe700 1 --2- 172.21.15.3:0/2137677463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f245000b5f0 tx=0x7f2450006ee0).ready entity=mon.0 client_cookie=fb5e7d02034bfc8e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f245e7fc700 1 -- 172.21.15.3:0/2137677463 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f245000d600 con 0x7f2460070370 2023-11-15T05:54:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f245e7fc700 1 -- 172.21.15.3:0/2137677463 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f245000dbd0 con 0x7f2460070370 2023-11-15T05:54:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f245e7fc700 1 -- 172.21.15.3:0/2137677463 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2450011c00 con 0x7f2460070370 2023-11-15T05:54:23.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f2465b1e700 1 -- 172.21.15.3:0/2137677463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 msgr2=0x7f24600fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f2465b1e700 1 --2- 172.21.15.3:0/2137677463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f245000b5f0 tx=0x7f2450006ee0).stop 2023-11-15T05:54:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f2465b1e700 1 -- 172.21.15.3:0/2137677463 shutdown_connections 2023-11-15T05:54:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f2465b1e700 1 --2- 172.21.15.3:0/2137677463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f24600fe9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.253+0000 7f2465b1e700 1 -- 172.21.15.3:0/2137677463 >> 172.21.15.3:0/2137677463 conn(0x7f24600f63c0 msgr2=0x7f24600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:23.266 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.254+0000 7f2465b1e700 1 -- 172.21.15.3:0/2137677463 shutdown_connections 2023-11-15T05:54:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.254+0000 7f2465b1e700 1 -- 172.21.15.3:0/2137677463 wait complete. 2023-11-15T05:54:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.254+0000 7f2465b1e700 1 Processor -- start 2023-11-15T05:54:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.254+0000 7f2465b1e700 1 -- start start 2023-11-15T05:54:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2465b1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:23.267 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2465b1e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246010d070 con 0x7f2460070370 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56368/0 (socket says 172.21.15.3:56368) 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f245f7fe700 1 -- 172.21.15.3:0/1911129389 learned_addr learned my addr 172.21.15.3:0/1911129389 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f245f7fe700 1 -- 172.21.15.3:0/1911129389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245000b200 con 0x7f2460070370 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f245f7fe700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f245000b190 tx=0x7f2450006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f245001b410 con 0x7f2460070370 2023-11-15T05:54:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f245001b9e0 con 0x7f2460070370 2023-11-15T05:54:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246010d270 con 0x7f2460070370 2023-11-15T05:54:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f245001a8f0 con 0x7f2460070370 2023-11-15T05:54:23.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.255+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246010d680 con 0x7f2460070370 2023-11-15T05:54:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.256+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f245000f070 con 0x7f2460070370 2023-11-15T05:54:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.256+0000 7f2447fff700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 0x7f244c0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.256+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2450041360 con 0x7f2460070370 2023-11-15T05:54:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.256+0000 7f245effd700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 0x7f244c0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:23.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.257+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460045ad0 con 0x7f2460070370 2023-11-15T05:54:23.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.257+0000 7f245effd700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 0x7f244c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:23.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.261+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2450016070 con 0x7f2460070370 2023-11-15T05:54:23.457 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.455+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 --> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f2460106c90 con 0x7f244c02fcf0 2023-11-15T05:54:23.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.459+0000 7f2447fff700 1 -- 172.21.15.3:0/1911129389 <== mgr.14144 v2:172.21.15.3:6800/2695659223 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2460106c90 con 0x7f244c02fcf0 2023-11-15T05:54:23.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T05:54:23.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 msgr2=0x7f244c0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:23.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 0x7f244c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24540096a0 tx=0x7f2454006b40).stop 2023-11-15T05:54:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 msgr2=0x7f246010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f245000b190 tx=0x7f2450006ee0).stop 2023-11-15T05:54:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 shutdown_connections 2023-11-15T05:54:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f244c02fcf0 0x7f244c0321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.461+0000 7f2465b1e700 1 --2- 172.21.15.3:0/1911129389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2460070370 0x7f246010cb80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:23.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.462+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 >> 172.21.15.3:0/1911129389 conn(0x7f24600f63c0 msgr2=0x7f24600f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:23.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.462+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 shutdown_connections 2023-11-15T05:54:23.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:23.462+0000 7f2465b1e700 1 -- 172.21.15.3:0/1911129389 wait complete. 2023-11-15T05:54:23.796 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:22.497936+0000 mon.smithi003 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi003.cldzke(active, since 1.36896s) 2023-11-15T05:54:23.797 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:22.499455+0000 mgr.smithi003.cldzke (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T05:54:23.797 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:22.501802+0000 mon.smithi003 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:23.797 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:23.461246+0000 mon.smithi003 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:23.826 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying node-exporter service with default placement... 2023-11-15T05:54:23.826 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T05:54:24.201 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.197+0000 7fbc6407f700 1 Processor -- start 2023-11-15T05:54:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.197+0000 7fbc6407f700 1 -- start start 2023-11-15T05:54:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.197+0000 7fbc6407f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.197+0000 7fbc6407f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0fb490 con 0x7fbc5c0fab80 2023-11-15T05:54:24.202 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.198+0000 7fbc61e1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.198+0000 7fbc61e1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56374/0 (socket says 172.21.15.3:56374) 2023-11-15T05:54:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.198+0000 7fbc61e1b700 1 -- 172.21.15.3:0/1892394039 learned_addr learned my addr 172.21.15.3:0/1892394039 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.198+0000 7fbc61e1b700 1 -- 172.21.15.3:0/1892394039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0fb5d0 con 0x7fbc5c0fab80 2023-11-15T05:54:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.198+0000 7fbc61e1b700 1 --2- 172.21.15.3:0/1892394039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbc5800b5f0 tx=0x7fbc58006ee0).ready entity=mon.0 client_cookie=dcf953a2f5e20945 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.199+0000 7fbc60e19700 1 -- 172.21.15.3:0/1892394039 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc5800d600 con 0x7fbc5c0fab80 2023-11-15T05:54:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.199+0000 7fbc60e19700 1 -- 172.21.15.3:0/1892394039 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbc5800dbd0 con 0x7fbc5c0fab80 2023-11-15T05:54:24.204 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.199+0000 7fbc60e19700 1 -- 172.21.15.3:0/1892394039 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc58011c00 con 0x7fbc5c0fab80 2023-11-15T05:54:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 -- 172.21.15.3:0/1892394039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 msgr2=0x7fbc5c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 --2- 172.21.15.3:0/1892394039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbc5800b5f0 tx=0x7fbc58006ee0).stop 2023-11-15T05:54:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 -- 172.21.15.3:0/1892394039 shutdown_connections 2023-11-15T05:54:24.207 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 --2- 172.21.15.3:0/1892394039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c0fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 -- 172.21.15.3:0/1892394039 >> 172.21.15.3:0/1892394039 conn(0x7fbc5c0f63c0 msgr2=0x7fbc5c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 -- 172.21.15.3:0/1892394039 shutdown_connections 2023-11-15T05:54:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 -- 172.21.15.3:0/1892394039 wait complete. 2023-11-15T05:54:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.200+0000 7fbc6407f700 1 Processor -- start 2023-11-15T05:54:24.208 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc6407f700 1 -- start start 2023-11-15T05:54:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc6407f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc6407f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c10aef0 con 0x7fbc5c0fab80 2023-11-15T05:54:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc61e1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc61e1b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56390/0 (socket says 172.21.15.3:56390) 2023-11-15T05:54:24.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc61e1b700 1 -- 172.21.15.3:0/2201006130 learned_addr learned my addr 172.21.15.3:0/2201006130 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:24.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc61e1b700 1 -- 172.21.15.3:0/2201006130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5800b200 con 0x7fbc5c0fab80 2023-11-15T05:54:24.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc61e1b700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbc5800b1d0 tx=0x7fbc58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:24.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc5801b410 con 0x7fbc5c0fab80 2023-11-15T05:54:24.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbc5801b9e0 con 0x7fbc5c0fab80 2023-11-15T05:54:24.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc5801a840 con 0x7fbc5c0fab80 2023-11-15T05:54:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.201+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c10b0f0 con 0x7fbc5c0fab80 2023-11-15T05:54:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.202+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c10b500 con 0x7fbc5c0fab80 2023-11-15T05:54:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.202+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbc5800f070 con 0x7fbc5c0fab80 2023-11-15T05:54:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.203+0000 7fbc4a7fc700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 0x7fbc44032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.203+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbc58041370 con 0x7fbc5c0fab80 2023-11-15T05:54:24.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.203+0000 7fbc6161a700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 0x7fbc44032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:24.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.203+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c059100 con 0x7fbc5c0fab80 2023-11-15T05:54:24.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.203+0000 7fbc6161a700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 0x7fbc44032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:24.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.207+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc58016070 con 0x7fbc5c0fab80 2023-11-15T05:54:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.404+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 --> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc5c1049a0 con 0x7fbc4402feb0 2023-11-15T05:54:24.411 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.409+0000 7fbc4a7fc700 1 -- 172.21.15.3:0/2201006130 <== mgr.14144 v2:172.21.15.3:6800/2695659223 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fbc5c1049a0 con 0x7fbc4402feb0 2023-11-15T05:54:24.411 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T05:54:24.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 msgr2=0x7fbc44032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:24.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 0x7fbc44032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).stop 2023-11-15T05:54:24.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 msgr2=0x7fbc5c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:24.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbc5800b1d0 tx=0x7fbc58006ee0).stop 2023-11-15T05:54:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 shutdown_connections 2023-11-15T05:54:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fbc4402feb0 0x7fbc44032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 --2- 172.21.15.3:0/2201006130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc5c0fab80 0x7fbc5c10aa00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 >> 172.21.15.3:0/2201006130 conn(0x7fbc5c0f63c0 msgr2=0x7fbc5c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 shutdown_connections 2023-11-15T05:54:24.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:24.411+0000 7fbc6407f700 1 -- 172.21.15.3:0/2201006130 wait complete. 2023-11-15T05:54:24.692 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:23.458167+0000 mgr.smithi003.cldzke (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:24.692 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:23.458742+0000 mgr.smithi003.cldzke (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T05:54:24.693 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:23.502667+0000 mon.smithi003 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi003.cldzke(active, since 2s) 2023-11-15T05:54:24.693 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:24.410356+0000 mon.smithi003 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:24.826 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying alertmanager service with default placement... 2023-11-15T05:54:24.827 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T05:54:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e3681700 1 Processor -- start 2023-11-15T05:54:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e3681700 1 -- start start 2023-11-15T05:54:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e3681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e3681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0fedd0 con 0x7fd4dc070430 2023-11-15T05:54:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e141d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e141d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60648/0 (socket says 172.21.15.3:60648) 2023-11-15T05:54:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.207+0000 7fd4e141d700 1 -- 172.21.15.3:0/261724554 learned_addr learned my addr 172.21.15.3:0/261724554 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4e141d700 1 -- 172.21.15.3:0/261724554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0fef10 con 0x7fd4dc070430 2023-11-15T05:54:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4e141d700 1 --2- 172.21.15.3:0/261724554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd4d800b5f0 tx=0x7fd4d8006ee0).ready entity=mon.0 client_cookie=a158dae71bb8bf0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4cbfff700 1 -- 172.21.15.3:0/261724554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d800d600 con 0x7fd4dc070430 2023-11-15T05:54:25.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4cbfff700 1 -- 172.21.15.3:0/261724554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4d800dbd0 con 0x7fd4dc070430 2023-11-15T05:54:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4cbfff700 1 -- 172.21.15.3:0/261724554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d8011c00 con 0x7fd4dc070430 2023-11-15T05:54:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4e3681700 1 -- 172.21.15.3:0/261724554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 msgr2=0x7fd4dc0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.208+0000 7fd4e3681700 1 --2- 172.21.15.3:0/261724554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd4d800b5f0 tx=0x7fd4d8006ee0).stop 2023-11-15T05:54:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 -- 172.21.15.3:0/261724554 shutdown_connections 2023-11-15T05:54:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 --2- 172.21.15.3:0/261724554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc0fe8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 -- 172.21.15.3:0/261724554 >> 172.21.15.3:0/261724554 conn(0x7fd4dc0f63c0 msgr2=0x7fd4dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 -- 172.21.15.3:0/261724554 shutdown_connections 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 -- 172.21.15.3:0/261724554 wait complete. 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 Processor -- start 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.209+0000 7fd4e3681700 1 -- start start 2023-11-15T05:54:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e3681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e3681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc10aef0 con 0x7fd4dc070430 2023-11-15T05:54:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e141d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e141d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60664/0 (socket says 172.21.15.3:60664) 2023-11-15T05:54:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e141d700 1 -- 172.21.15.3:0/801112049 learned_addr learned my addr 172.21.15.3:0/801112049 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e141d700 1 -- 172.21.15.3:0/801112049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d800b200 con 0x7fd4dc070430 2023-11-15T05:54:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e141d700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd4d800b190 tx=0x7fd4d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d801b410 con 0x7fd4dc070430 2023-11-15T05:54:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4d801b9e0 con 0x7fd4dc070430 2023-11-15T05:54:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d801a880 con 0x7fd4dc070430 2023-11-15T05:54:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.210+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc10b0f0 con 0x7fd4dc070430 2023-11-15T05:54:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.211+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc10b500 con 0x7fd4dc070430 2023-11-15T05:54:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.211+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd4d800f070 con 0x7fd4dc070430 2023-11-15T05:54:25.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.212+0000 7fd4c9ffb700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 0x7fd4c4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.212+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd4d8041380 con 0x7fd4dc070430 2023-11-15T05:54:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.212+0000 7fd4e0c1c700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 0x7fd4c4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.212+0000 7fd4e0c1c700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 0x7fd4c4032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4cc0096a0 tx=0x7fd4cc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.212+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc059100 con 0x7fd4dc070430 2023-11-15T05:54:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.217+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4d8025460 con 0x7fd4dc070430 2023-11-15T05:54:25.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.413+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 --> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4dc1048f0 con 0x7fd4c402feb0 2023-11-15T05:54:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.418+0000 7fd4c9ffb700 1 -- 172.21.15.3:0/801112049 <== mgr.14144 v2:172.21.15.3:6800/2695659223 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd4dc1048f0 con 0x7fd4c402feb0 2023-11-15T05:54:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T05:54:25.422 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 msgr2=0x7fd4c4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:25.422 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 0x7fd4c4032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4cc0096a0 tx=0x7fd4cc006b40).stop 2023-11-15T05:54:25.422 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 msgr2=0x7fd4dc10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd4d800b190 tx=0x7fd4d8006ee0).stop 2023-11-15T05:54:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 shutdown_connections 2023-11-15T05:54:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7fd4c402feb0 0x7fd4c4032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 --2- 172.21.15.3:0/801112049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4dc070430 0x7fd4dc10aa00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.419+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 >> 172.21.15.3:0/801112049 conn(0x7fd4dc0f63c0 msgr2=0x7fd4dc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:25.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.420+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 shutdown_connections 2023-11-15T05:54:25.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:25.420+0000 7fd4e3681700 1 -- 172.21.15.3:0/801112049 wait complete. 2023-11-15T05:54:25.729 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:24.407175+0000 mgr.smithi003.cldzke (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:25.729 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:24.407891+0000 mgr.smithi003.cldzke (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T05:54:25.729 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:25.419313+0000 mon.smithi003 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.3:0/3020860873' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T05:54:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.146+0000 7f950f59b700 1 Processor -- start 2023-11-15T05:54:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.146+0000 7f950f59b700 1 -- start start 2023-11-15T05:54:26.150 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.146+0000 7f950f59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:26.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.146+0000 7f950f59b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080fc9e0 con 0x7f95080fc0d0 2023-11-15T05:54:26.151 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950e599700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950e599700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60666/0 (socket says 172.21.15.3:60666) 2023-11-15T05:54:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950e599700 1 -- 172.21.15.3:0/3035497560 learned_addr learned my addr 172.21.15.3:0/3035497560 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:26.152 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950e599700 1 -- 172.21.15.3:0/3035497560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080fcb20 con 0x7f95080fc0d0 2023-11-15T05:54:26.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950e599700 1 --2- 172.21.15.3:0/3035497560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f950400b5f0 tx=0x7f9504006ee0).ready entity=mon.0 client_cookie=efc601d5a230904c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:26.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950d597700 1 -- 172.21.15.3:0/3035497560 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950400d600 con 0x7f95080fc0d0 2023-11-15T05:54:26.153 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950d597700 1 -- 172.21.15.3:0/3035497560 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f950400dbd0 con 0x7f95080fc0d0 2023-11-15T05:54:26.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.147+0000 7f950d597700 1 -- 172.21.15.3:0/3035497560 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9504011c00 con 0x7f95080fc0d0 2023-11-15T05:54:26.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 -- 172.21.15.3:0/3035497560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 msgr2=0x7f95080fc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:26.154 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 --2- 172.21.15.3:0/3035497560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f950400b5f0 tx=0x7f9504006ee0).stop 2023-11-15T05:54:26.160 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 -- 172.21.15.3:0/3035497560 shutdown_connections 2023-11-15T05:54:26.160 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 --2- 172.21.15.3:0/3035497560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f95080fc4f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:26.160 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 -- 172.21.15.3:0/3035497560 >> 172.21.15.3:0/3035497560 conn(0x7f95080f59a0 msgr2=0x7f95080f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 -- 172.21.15.3:0/3035497560 shutdown_connections 2023-11-15T05:54:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.148+0000 7f950f59b700 1 -- 172.21.15.3:0/3035497560 wait complete. 2023-11-15T05:54:26.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950f59b700 1 Processor -- start 2023-11-15T05:54:26.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950f59b700 1 -- start start 2023-11-15T05:54:26.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950f59b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950f59b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950810cfc0 con 0x7f95080fc0d0 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950e599700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950e599700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60676/0 (socket says 172.21.15.3:60676) 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.149+0000 7f950e599700 1 -- 172.21.15.3:0/557836607 learned_addr learned my addr 172.21.15.3:0/557836607 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f950e599700 1 -- 172.21.15.3:0/557836607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950400b200 con 0x7f95080fc0d0 2023-11-15T05:54:26.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f950e599700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f950400b1d0 tx=0x7f9504006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:26.164 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9504011690 con 0x7f95080fc0d0 2023-11-15T05:54:26.164 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f950401b880 con 0x7f95080fc0d0 2023-11-15T05:54:26.164 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950810d1c0 con 0x7f95080fc0d0 2023-11-15T05:54:26.165 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f950401a650 con 0x7f95080fc0d0 2023-11-15T05:54:26.165 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.150+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950810d5d0 con 0x7f95080fc0d0 2023-11-15T05:54:26.165 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.151+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f950400f070 con 0x7f95080fc0d0 2023-11-15T05:54:26.165 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.151+0000 7f94faffd700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 0x7f94f0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:26.166 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.151+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9504041140 con 0x7f95080fc0d0 2023-11-15T05:54:26.166 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.151+0000 7f950dd98700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 0x7f94f0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.152+0000 7f950dd98700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 0x7f94f0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f94fc0096a0 tx=0x7f94fc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.152+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9508059100 con 0x7f95080fc0d0 2023-11-15T05:54:26.167 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.156+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9504016070 con 0x7f95080fc0d0 2023-11-15T05:54:26.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.322+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f95081068d0 con 0x7f95080fc0d0 2023-11-15T05:54:26.327 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.325+0000 7f94faffd700 1 -- 172.21.15.3:0/557836607 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9504022720 con 0x7f95080fc0d0 2023-11-15T05:54:26.330 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 msgr2=0x7f94f0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:26.330 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 0x7f94f0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f94fc0096a0 tx=0x7f94fc006b40).stop 2023-11-15T05:54:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 msgr2=0x7f950810cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f950400b1d0 tx=0x7f9504006ee0).stop 2023-11-15T05:54:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 shutdown_connections 2023-11-15T05:54:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f94f002feb0 0x7f94f0032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 --2- 172.21.15.3:0/557836607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95080fc0d0 0x7f950810cad0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 >> 172.21.15.3:0/557836607 conn(0x7f95080f59a0 msgr2=0x7f95080f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 shutdown_connections 2023-11-15T05:54:26.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:26.327+0000 7f950f59b700 1 -- 172.21.15.3:0/557836607 wait complete. 2023-11-15T05:54:26.645 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:25.416229+0000 mgr.smithi003.cldzke (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:26.646 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:25.416801+0000 mgr.smithi003.cldzke (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T05:54:26.646 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:26.326978+0000 mon.smithi003 (mon.0) 83 : audit [INF] from='client.? 172.21.15.3:0/557836607' entity='client.admin' 2023-11-15T05:54:26.685 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T05:54:27.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.057+0000 7f2be0d71700 1 Processor -- start 2023-11-15T05:54:27.060 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.057+0000 7f2be0d71700 1 -- start start 2023-11-15T05:54:27.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2be0d71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:27.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2be0d71700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc0fc730 con 0x7f2bdc0703d0 2023-11-15T05:54:27.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2bda59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:27.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2bda59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60682/0 (socket says 172.21.15.3:60682) 2023-11-15T05:54:27.061 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2bda59c700 1 -- 172.21.15.3:0/2681253579 learned_addr learned my addr 172.21.15.3:0/2681253579 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:27.064 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2bda59c700 1 -- 172.21.15.3:0/2681253579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc101230 con 0x7f2bdc0703d0 2023-11-15T05:54:27.064 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.058+0000 7f2bda59c700 1 --2- 172.21.15.3:0/2681253579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2bcc00b5f0 tx=0x7f2bcc006ee0).ready entity=mon.0 client_cookie=94a6fa8eea7dfe5c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:27.064 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2bd959a700 1 -- 172.21.15.3:0/2681253579 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc00d600 con 0x7f2bdc0703d0 2023-11-15T05:54:27.064 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2bd959a700 1 -- 172.21.15.3:0/2681253579 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2bcc00dbd0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2bd959a700 1 -- 172.21.15.3:0/2681253579 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc011c00 con 0x7f2bdc0703d0 2023-11-15T05:54:27.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 -- 172.21.15.3:0/2681253579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 msgr2=0x7f2bdc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:27.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 --2- 172.21.15.3:0/2681253579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2bcc00b5f0 tx=0x7f2bcc006ee0).stop 2023-11-15T05:54:27.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 -- 172.21.15.3:0/2681253579 shutdown_connections 2023-11-15T05:54:27.065 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 --2- 172.21.15.3:0/2681253579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc100d40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:27.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 -- 172.21.15.3:0/2681253579 >> 172.21.15.3:0/2681253579 conn(0x7f2bdc0f63c0 msgr2=0x7f2bdc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:27.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.059+0000 7f2be0d71700 1 -- 172.21.15.3:0/2681253579 shutdown_connections 2023-11-15T05:54:27.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.060+0000 7f2be0d71700 1 -- 172.21.15.3:0/2681253579 wait complete. 2023-11-15T05:54:27.066 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.060+0000 7f2be0d71700 1 Processor -- start 2023-11-15T05:54:27.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.060+0000 7f2be0d71700 1 -- start start 2023-11-15T05:54:27.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2be0d71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:27.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2be0d71700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc10d0a0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.067 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bda59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:27.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bda59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60688/0 (socket says 172.21.15.3:60688) 2023-11-15T05:54:27.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bda59c700 1 -- 172.21.15.3:0/338537319 learned_addr learned my addr 172.21.15.3:0/338537319 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:27.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bda59c700 1 -- 172.21.15.3:0/338537319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc00b200 con 0x7f2bdc0703d0 2023-11-15T05:54:27.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bda59c700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2bcc00b190 tx=0x7f2bcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:27.068 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc01b410 con 0x7f2bdc0703d0 2023-11-15T05:54:27.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2bcc01b9e0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc01a880 con 0x7f2bdc0703d0 2023-11-15T05:54:27.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.061+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc10d2a0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.070 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.062+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc10d6b0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.071 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.062+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2bcc00f070 con 0x7f2bdc0703d0 2023-11-15T05:54:27.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.062+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc045ad0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.062+0000 7f2bcaffd700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 0x7f2bc0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:27.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.063+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2bcc041350 con 0x7f2bdc0703d0 2023-11-15T05:54:27.072 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.063+0000 7f2bd9d9b700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 0x7f2bc0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.063+0000 7f2bd9d9b700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 0x7f2bc0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f2bd00096a0 tx=0x7f2bd0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.067+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bcc0226d0 con 0x7f2bdc0703d0 2023-11-15T05:54:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.232+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f2bdc101d90 con 0x7f2bdc0703d0 2023-11-15T05:54:27.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.235+0000 7f2bcaffd700 1 -- 172.21.15.3:0/338537319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2bcc016070 con 0x7f2bdc0703d0 2023-11-15T05:54:27.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 msgr2=0x7f2bc0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:27.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 0x7f2bc0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f2bd00096a0 tx=0x7f2bd0006b40).stop 2023-11-15T05:54:27.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 msgr2=0x7f2bdc10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:27.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2bcc00b190 tx=0x7f2bcc006ee0).stop 2023-11-15T05:54:27.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 shutdown_connections 2023-11-15T05:54:27.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f2bc002feb0 0x7f2bc0032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:27.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 --2- 172.21.15.3:0/338537319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bdc0703d0 0x7f2bdc10cbb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:27.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.236+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 >> 172.21.15.3:0/338537319 conn(0x7f2bdc0f63c0 msgr2=0x7f2bdc0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:27.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.237+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 shutdown_connections 2023-11-15T05:54:27.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:27.237+0000 7f2be0d71700 1 -- 172.21.15.3:0/338537319 wait complete. 2023-11-15T05:54:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling the dashboard module... 2023-11-15T05:54:27.810 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T05:54:28.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.206+0000 7f150b59e700 1 Processor -- start 2023-11-15T05:54:28.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.206+0000 7f150b59e700 1 -- start start 2023-11-15T05:54:28.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.206+0000 7f150b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:28.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.206+0000 7f150b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c0fb2e0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.212 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60696/0 (socket says 172.21.15.3:60696) 2023-11-15T05:54:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150a59c700 1 -- 172.21.15.3:0/1096957729 learned_addr learned my addr 172.21.15.3:0/1096957729 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150a59c700 1 -- 172.21.15.3:0/1096957729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c0fb420 con 0x7f150c0fa9d0 2023-11-15T05:54:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150a59c700 1 --2- 172.21.15.3:0/1096957729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14fc00b870 tx=0x7f14fc006f70).ready entity=mon.0 client_cookie=c624508d5bbacb7d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150959a700 1 -- 172.21.15.3:0/1096957729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc00da20 con 0x7f150c0fa9d0 2023-11-15T05:54:28.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150959a700 1 -- 172.21.15.3:0/1096957729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f14fc010470 con 0x7f150c0fa9d0 2023-11-15T05:54:28.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.207+0000 7f150959a700 1 -- 172.21.15.3:0/1096957729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc00f5b0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 -- 172.21.15.3:0/1096957729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 msgr2=0x7f150c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 --2- 172.21.15.3:0/1096957729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14fc00b870 tx=0x7f14fc006f70).stop 2023-11-15T05:54:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 -- 172.21.15.3:0/1096957729 shutdown_connections 2023-11-15T05:54:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 --2- 172.21.15.3:0/1096957729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c0fadf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 -- 172.21.15.3:0/1096957729 >> 172.21.15.3:0/1096957729 conn(0x7f150c0f6210 msgr2=0x7f150c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:28.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 -- 172.21.15.3:0/1096957729 shutdown_connections 2023-11-15T05:54:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.208+0000 7f150b59e700 1 -- 172.21.15.3:0/1096957729 wait complete. 2023-11-15T05:54:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150b59e700 1 Processor -- start 2023-11-15T05:54:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150b59e700 1 -- start start 2023-11-15T05:54:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:28.216 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c108a60 con 0x7f150c0fa9d0 2023-11-15T05:54:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60712/0 (socket says 172.21.15.3:60712) 2023-11-15T05:54:28.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150a59c700 1 -- 172.21.15.3:0/3601120564 learned_addr learned my addr 172.21.15.3:0/3601120564 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.209+0000 7f150a59c700 1 -- 172.21.15.3:0/3601120564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc00b480 con 0x7f150c0fa9d0 2023-11-15T05:54:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f150a59c700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f14fc006800 tx=0x7f14fc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc00f820 con 0x7f150c0fa9d0 2023-11-15T05:54:28.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f14fc00fdf0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c108c60 con 0x7f150c0fa9d0 2023-11-15T05:54:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14fc019d50 con 0x7f150c0fa9d0 2023-11-15T05:54:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.210+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c109070 con 0x7f150c0fa9d0 2023-11-15T05:54:28.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.211+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f14fc016030 con 0x7f150c0fa9d0 2023-11-15T05:54:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.211+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c045ad0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.212+0000 7f1502ffd700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 0x7f14ec032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.212+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f14fc043ea0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.212+0000 7f1509d9b700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 0x7f14ec032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:28.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.212+0000 7f1509d9b700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 0x7f14ec032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f14f40096a0 tx=0x7f14f4006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:28.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.216+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14fc02c950 con 0x7f150c0fa9d0 2023-11-15T05:54:28.428 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:28.426+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f150c0fc6e0 con 0x7f150c0fa9d0 2023-11-15T05:54:28.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:27.236601+0000 mon.smithi003 (mon.0) 84 : audit [INF] from='client.? 172.21.15.3:0/338537319' entity='client.admin' 2023-11-15T05:54:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.238+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f14fc024420 con 0x7f150c0fa9d0 2023-11-15T05:54:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.239+0000 7f1502ffd700 1 -- 172.21.15.3:0/3601120564 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f14fc0446a0 con 0x7f150c0fa9d0 2023-11-15T05:54:29.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 msgr2=0x7f14ec032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:29.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 0x7f14ec032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f14f40096a0 tx=0x7f14f4006b40).stop 2023-11-15T05:54:29.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 msgr2=0x7f150c108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:29.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f14fc006800 tx=0x7f14fc006f70).stop 2023-11-15T05:54:29.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 shutdown_connections 2023-11-15T05:54:29.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f14ec02fe60 0x7f14ec032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:29.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 --2- 172.21.15.3:0/3601120564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f150c0fa9d0 0x7f150c108570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:29.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 >> 172.21.15.3:0/3601120564 conn(0x7f150c0f6210 msgr2=0x7f150c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:29.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 shutdown_connections 2023-11-15T05:54:29.248 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:29.243+0000 7f150b59e700 1 -- 172.21.15.3:0/3601120564 wait complete. 2023-11-15T05:54:29.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:29 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:28.428414+0000 mon.smithi003 (mon.0) 85 : audit [INF] from='client.? 172.21.15.3:0/3601120564' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T05:54:29.692 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T05:54:30.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.236+0000 7f13d13c8700 1 Processor -- start 2023-11-15T05:54:30.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.236+0000 7f13d13c8700 1 -- start start 2023-11-15T05:54:30.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.236+0000 7f13d13c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:30.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.236+0000 7f13d13c8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0fc750 con 0x7f13cc0703b0 2023-11-15T05:54:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60732/0 (socket says 172.21.15.3:60732) 2023-11-15T05:54:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13caffd700 1 -- 172.21.15.3:0/3405535603 learned_addr learned my addr 172.21.15.3:0/3405535603 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13caffd700 1 -- 172.21.15.3:0/3405535603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc101250 con 0x7f13cc0703b0 2023-11-15T05:54:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13caffd700 1 --2- 172.21.15.3:0/3405535603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f13bc00b5f0 tx=0x7f13bc006ee0).ready entity=mon.0 client_cookie=aecce2625f149877 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13c9ffb700 1 -- 172.21.15.3:0/3405535603 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc00d600 con 0x7f13cc0703b0 2023-11-15T05:54:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13c9ffb700 1 -- 172.21.15.3:0/3405535603 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f13bc00dbd0 con 0x7f13cc0703b0 2023-11-15T05:54:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.237+0000 7f13c9ffb700 1 -- 172.21.15.3:0/3405535603 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc011c00 con 0x7f13cc0703b0 2023-11-15T05:54:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.238+0000 7f13d13c8700 1 -- 172.21.15.3:0/3405535603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 msgr2=0x7f13cc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.238+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3405535603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f13bc00b5f0 tx=0x7f13bc006ee0).stop 2023-11-15T05:54:30.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.239+0000 7f13d13c8700 1 -- 172.21.15.3:0/3405535603 shutdown_connections 2023-11-15T05:54:30.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.239+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3405535603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc100d60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:30.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.239+0000 7f13d13c8700 1 -- 172.21.15.3:0/3405535603 >> 172.21.15.3:0/3405535603 conn(0x7f13cc0f63c0 msgr2=0x7f13cc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:30.244 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.239+0000 7f13d13c8700 1 -- 172.21.15.3:0/3405535603 shutdown_connections 2023-11-15T05:54:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.239+0000 7f13d13c8700 1 -- 172.21.15.3:0/3405535603 wait complete. 2023-11-15T05:54:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.240+0000 7f13d13c8700 1 Processor -- start 2023-11-15T05:54:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.240+0000 7f13d13c8700 1 -- start start 2023-11-15T05:54:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.240+0000 7f13d13c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:30.245 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.240+0000 7f13d13c8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc10d100 con 0x7f13cc0703b0 2023-11-15T05:54:30.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:30.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13caffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60742/0 (socket says 172.21.15.3:60742) 2023-11-15T05:54:30.246 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13caffd700 1 -- 172.21.15.3:0/3139162843 learned_addr learned my addr 172.21.15.3:0/3139162843 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13caffd700 1 -- 172.21.15.3:0/3139162843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc00b200 con 0x7f13cc0703b0 2023-11-15T05:54:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13caffd700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13bc00b190 tx=0x7f13bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:30.247 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc01b410 con 0x7f13cc0703b0 2023-11-15T05:54:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f13bc01b9e0 con 0x7f13cc0703b0 2023-11-15T05:54:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc10d300 con 0x7f13cc0703b0 2023-11-15T05:54:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13bc01a880 con 0x7f13cc0703b0 2023-11-15T05:54:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.241+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc10d710 con 0x7f13cc0703b0 2023-11-15T05:54:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc045ad0 con 0x7f13cc0703b0 2023-11-15T05:54:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f13bc00f070 con 0x7f13cc0703b0 2023-11-15T05:54:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13b37fe700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 0x7f13b8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f13bc041450 con 0x7f13cc0703b0 2023-11-15T05:54:30.250 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13ca7fc700 1 -- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 msgr2=0x7f13b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:30.250 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.243+0000 7f13ca7fc700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 0x7f13b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:54:30.250 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.248+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13bc019cd0 con 0x7f13cc0703b0 2023-11-15T05:54:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.444+0000 7f13ca7fc700 1 -- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 msgr2=0x7f13b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:30.446 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.444+0000 7f13ca7fc700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 0x7f13b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:54:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.456+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f13cc10e7c0 con 0x7f13cc0703b0 2023-11-15T05:54:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.457+0000 7f13b37fe700 1 -- 172.21.15.3:0/3139162843 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f13bc016070 con 0x7f13cc0703b0 2023-11-15T05:54:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:54:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T05:54:30.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-15T05:54:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "active_name": "smithi003.cldzke", 2023-11-15T05:54:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T05:54:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:54:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 msgr2=0x7f13b8032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:54:30.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 0x7f13b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 msgr2=0x7f13cc10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f13bc00b190 tx=0x7f13bc006ee0).stop 2023-11-15T05:54:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 shutdown_connections 2023-11-15T05:54:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f13b802feb0 0x7f13b8032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 --2- 172.21.15.3:0/3139162843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f13cc0703b0 0x7f13cc10cc10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.459+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 >> 172.21.15.3:0/3139162843 conn(0x7f13cc0f63c0 msgr2=0x7f13cc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.460+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 shutdown_connections 2023-11-15T05:54:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:30.460+0000 7f13d13c8700 1 -- 172.21.15.3:0/3139162843 wait complete. 2023-11-15T05:54:30.491 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:29.241521+0000 mon.smithi003 (mon.0) 86 : audit [INF] from='client.? 172.21.15.3:0/3601120564' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T05:54:30.491 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:29.241597+0000 mon.smithi003 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi003.cldzke(active, since 8s) 2023-11-15T05:54:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for the mgr to restart... 2023-11-15T05:54:30.867 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr epoch 13... 2023-11-15T05:54:30.868 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T05:54:31.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04f5b9d700 1 Processor -- start 2023-11-15T05:54:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04f5b9d700 1 -- start start 2023-11-15T05:54:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04f5b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:31.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04f5b9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00fb490 con 0x7f04f00fab80 2023-11-15T05:54:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60750/0 (socket says 172.21.15.3:60750) 2023-11-15T05:54:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.220+0000 7f04ef7fe700 1 -- 172.21.15.3:0/176289360 learned_addr learned my addr 172.21.15.3:0/176289360 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.221+0000 7f04ef7fe700 1 -- 172.21.15.3:0/176289360 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00fb5d0 con 0x7f04f00fab80 2023-11-15T05:54:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.221+0000 7f04ef7fe700 1 --2- 172.21.15.3:0/176289360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f04e000b5f0 tx=0x7f04e0006ee0).ready entity=mon.0 client_cookie=ae4edcbbbdc280af server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.221+0000 7f04ee7fc700 1 -- 172.21.15.3:0/176289360 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e000d600 con 0x7f04f00fab80 2023-11-15T05:54:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.221+0000 7f04ee7fc700 1 -- 172.21.15.3:0/176289360 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f04e000dbd0 con 0x7f04f00fab80 2023-11-15T05:54:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.221+0000 7f04ee7fc700 1 -- 172.21.15.3:0/176289360 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e0011c00 con 0x7f04f00fab80 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 -- 172.21.15.3:0/176289360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 msgr2=0x7f04f00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/176289360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f04e000b5f0 tx=0x7f04e0006ee0).stop 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 -- 172.21.15.3:0/176289360 shutdown_connections 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/176289360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f00fafa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 -- 172.21.15.3:0/176289360 >> 172.21.15.3:0/176289360 conn(0x7f04f00f63c0 msgr2=0x7f04f00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 -- 172.21.15.3:0/176289360 shutdown_connections 2023-11-15T05:54:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.222+0000 7f04f5b9d700 1 -- 172.21.15.3:0/176289360 wait complete. 2023-11-15T05:54:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04f5b9d700 1 Processor -- start 2023-11-15T05:54:31.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04f5b9d700 1 -- start start 2023-11-15T05:54:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04f5b9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04f5b9d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f010aef0 con 0x7f04f00fab80 2023-11-15T05:54:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60766/0 (socket says 172.21.15.3:60766) 2023-11-15T05:54:31.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.223+0000 7f04ef7fe700 1 -- 172.21.15.3:0/228283927 learned_addr learned my addr 172.21.15.3:0/228283927 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04ef7fe700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e000b200 con 0x7f04f00fab80 2023-11-15T05:54:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04ef7fe700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f04e000b1d0 tx=0x7f04e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e001b410 con 0x7f04f00fab80 2023-11-15T05:54:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f04e001b9e0 con 0x7f04f00fab80 2023-11-15T05:54:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f010b0f0 con 0x7f04f00fab80 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e001a840 con 0x7f04f00fab80 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.224+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f010b500 con 0x7f04f00fab80 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f04e000f070 con 0x7f04f00fab80 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04d7fff700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f04e0042260 con 0x7f04f00fab80 2023-11-15T05:54:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f0059100 con 0x7f04dc02ff00 2023-11-15T05:54:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04eeffd700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.225+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T05:54:31.428 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.426+0000 7f04eeffd700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:31.428 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.426+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T05:54:31.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:30.458950+0000 mon.smithi003 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.3:0/3139162843' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T05:54:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.827+0000 7f04eeffd700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:31.827+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T05:54:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:32.628+0000 7f04eeffd700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:32.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:32.628+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T05:54:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:34.229+0000 7f04eeffd700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2695659223 2023-11-15T05:54:34.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:34.229+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T05:54:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:34.957+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f04e00257e0 con 0x7f04f00fab80 2023-11-15T05:54:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:34.957+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 msgr2=0x7f04dc0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T05:54:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:34.957+0000 7f04d7fff700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.959+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f04e0042710 con 0x7f04f00fab80 2023-11-15T05:54:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.960+0000 7f04d7fff700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 0x7f04dc035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.960+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f0059100 con 0x7f04dc0339e0 2023-11-15T05:54:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.960+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 0x7f04dc035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.961+0000 7f04eeffd700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 0x7f04dc035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f04e4006fd0 tx=0x7f04e4006d70).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.962+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f04f0059100 con 0x7f04dc0339e0 2023-11-15T05:54:35.970 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.968+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f04f0045ad0 con 0x7f04dc0339e0 2023-11-15T05:54:35.971 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.969+0000 7f04d7fff700 1 -- 172.21.15.3:0/228283927 <== mgr.14164 v2:172.21.15.3:6800/4263843884 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f04f0045ad0 con 0x7f04dc0339e0 2023-11-15T05:54:35.974 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-15T05:54:35.974 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T05:54:35.974 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T05:54:35.974 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-15T05:54:35.975 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.969+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 msgr2=0x7f04dc035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:35.976 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.969+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 0x7f04dc035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f04e4006fd0 tx=0x7f04e4006d70).stop 2023-11-15T05:54:35.977 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.969+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 msgr2=0x7f04f010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:35.977 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.969+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f04e000b1d0 tx=0x7f04e0006ee0).stop 2023-11-15T05:54:35.977 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 shutdown_connections 2023-11-15T05:54:35.977 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f04dc0339e0 0x7f04dc035dd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:35.978 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04f00fab80 0x7f04f010aa00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:35.978 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 --2- 172.21.15.3:0/228283927 >> [v2:172.21.15.3:6800/2695659223,v1:172.21.15.3:6801/2695659223] conn(0x7f04dc02ff00 0x7f04dc0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T05:54:35.978 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 >> 172.21.15.3:0/228283927 conn(0x7f04f00f63c0 msgr2=0x7f04f0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:35.979 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 shutdown_connections 2023-11-15T05:54:35.979 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:35.970+0000 7f04f5b9d700 1 -- 172.21.15.3:0/228283927 wait complete. 2023-11-15T05:54:36.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:34.906709+0000 mon.smithi003 (mon.0) 89 : cluster [INF] Active manager daemon smithi003.cldzke restarted 2023-11-15T05:54:36.182 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:34.907159+0000 mon.smithi003 (mon.0) 90 : cluster [INF] Activating manager daemon smithi003.cldzke 2023-11-15T05:54:36.182 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:34.908933+0000 mon.smithi003 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T05:54:36.182 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:34.960388+0000 mon.smithi003 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi003.cldzke(active, starting, since 0.0533562s) 2023-11-15T05:54:36.183 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:34.961492+0000 mon.smithi003 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-15T05:54:36.183 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:34.962163+0000 mon.smithi003 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mgr metadata", "who": "smithi003.cldzke", "id": "smithi003.cldzke"}]: dispatch 2023-11-15T05:54:36.183 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:34.963252+0000 mon.smithi003 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T05:54:36.184 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:34.963432+0000 mon.smithi003 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T05:54:36.184 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:34.963593+0000 mon.smithi003 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T05:54:36.184 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:34.965969+0000 mon.smithi003 (mon.0) 98 : cluster [INF] Manager daemon smithi003.cldzke is now available 2023-11-15T05:54:36.185 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.079236+0000 mon.smithi003 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:36.185 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.081023+0000 mon.smithi003 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T05:54:36.186 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.101463+0000 mon.smithi003 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:54:36.186 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.111698+0000 mon.smithi003 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:54:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:mgr epoch 13 is available 2023-11-15T05:54:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:Generating a dashboard self-signed certificate... 2023-11-15T05:54:36.286 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T05:54:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.696+0000 7f1edaf41700 1 Processor -- start 2023-11-15T05:54:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.696+0000 7f1edaf41700 1 -- start start 2023-11-15T05:54:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1edaf41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1edaf41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed40fc710 con 0x7f1ed40703f0 2023-11-15T05:54:36.701 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1ed8cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:36.702 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1ed8cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38310/0 (socket says 172.21.15.3:38310) 2023-11-15T05:54:36.702 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1ed8cdd700 1 -- 172.21.15.3:0/3347939720 learned_addr learned my addr 172.21.15.3:0/3347939720 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:36.702 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1ed8cdd700 1 -- 172.21.15.3:0/3347939720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed4101210 con 0x7f1ed40703f0 2023-11-15T05:54:36.703 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.697+0000 7f1ed8cdd700 1 --2- 172.21.15.3:0/3347939720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ed000b5f0 tx=0x7f1ed0006ee0).ready entity=mon.0 client_cookie=f2b4cc0dacb1af7e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:36.703 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.698+0000 7f1ecb7fe700 1 -- 172.21.15.3:0/3347939720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed000d600 con 0x7f1ed40703f0 2023-11-15T05:54:36.703 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.698+0000 7f1ecb7fe700 1 -- 172.21.15.3:0/3347939720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ed000dbd0 con 0x7f1ed40703f0 2023-11-15T05:54:36.703 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.698+0000 7f1ecb7fe700 1 -- 172.21.15.3:0/3347939720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed0011c00 con 0x7f1ed40703f0 2023-11-15T05:54:36.703 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.698+0000 7f1edaf41700 1 -- 172.21.15.3:0/3347939720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 msgr2=0x7f1ed4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:36.704 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.698+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3347939720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1ed000b5f0 tx=0x7f1ed0006ee0).stop 2023-11-15T05:54:36.704 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 -- 172.21.15.3:0/3347939720 shutdown_connections 2023-11-15T05:54:36.704 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3347939720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed4100d20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:36.704 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 -- 172.21.15.3:0/3347939720 >> 172.21.15.3:0/3347939720 conn(0x7f1ed40f63c0 msgr2=0x7f1ed40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:36.704 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 -- 172.21.15.3:0/3347939720 shutdown_connections 2023-11-15T05:54:36.705 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 -- 172.21.15.3:0/3347939720 wait complete. 2023-11-15T05:54:36.705 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 Processor -- start 2023-11-15T05:54:36.705 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.699+0000 7f1edaf41700 1 -- start start 2023-11-15T05:54:36.705 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1edaf41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:36.705 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1edaf41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed410d1e0 con 0x7f1ed40703f0 2023-11-15T05:54:36.706 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ed8cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:36.707 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ed8cdd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38320/0 (socket says 172.21.15.3:38320) 2023-11-15T05:54:36.708 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ed8cdd700 1 -- 172.21.15.3:0/3860591582 learned_addr learned my addr 172.21.15.3:0/3860591582 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:36.708 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ed8cdd700 1 -- 172.21.15.3:0/3860591582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed000b200 con 0x7f1ed40703f0 2023-11-15T05:54:36.708 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ed8cdd700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ed000e040 tx=0x7f1ed0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:36.708 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.700+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed000d9f0 con 0x7f1ed40703f0 2023-11-15T05:54:36.708 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.701+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ed001a9f0 con 0x7f1ed40703f0 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.701+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed410d3e0 con 0x7f1ed40703f0 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.701+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed001b8c0 con 0x7f1ed40703f0 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.701+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed410d7f0 con 0x7f1ed40703f0 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.701+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f1ed000f070 con 0x7f1ed40703f0 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.702+0000 7f1ec97fa700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 0x7f1ec0032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.702+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ed00413e0 con 0x7f1ed40703f0 2023-11-15T05:54:36.710 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.702+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 0x7f1ec0032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:36.710 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.702+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed4101dd0 con 0x7f1ed40703f0 2023-11-15T05:54:36.710 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.702+0000 7f1ecbfff700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 0x7f1ec0032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ec40096a0 tx=0x7f1ec4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.707+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ed0016070 con 0x7f1ed40703f0 2023-11-15T05:54:36.880 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.878+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ed4101f90 con 0x7f1ec002fdc0 2023-11-15T05:54:36.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:36.963+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1ed00257e0 con 0x7f1ed40703f0 2023-11-15T05:54:37.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.188+0000 7f1ec97fa700 1 -- 172.21.15.3:0/3860591582 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1ed4101f90 con 0x7f1ec002fdc0 2023-11-15T05:54:37.190 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T05:54:37.191 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.189+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 msgr2=0x7f1ec0032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.189+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 0x7f1ec0032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ec40096a0 tx=0x7f1ec4006b40).stop 2023-11-15T05:54:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.189+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 msgr2=0x7f1ed410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.189+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f1ed000e040 tx=0x7f1ed0006ee0).stop 2023-11-15T05:54:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 shutdown_connections 2023-11-15T05:54:37.193 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1ec002fdc0 0x7f1ec0032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 --2- 172.21.15.3:0/3860591582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed40703f0 0x7f1ed410ccf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 >> 172.21.15.3:0/3860591582 conn(0x7f1ed40f63c0 msgr2=0x7f1ed40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:37.194 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 shutdown_connections 2023-11-15T05:54:37.195 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.190+0000 7f1edaf41700 1 -- 172.21.15.3:0/3860591582 wait complete. 2023-11-15T05:54:37.218 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.963357+0000 mgr.smithi003.cldzke (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T05:54:37.219 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:35.963870+0000 mon.smithi003 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi003.cldzke(active, since 1.05684s) 2023-11-15T05:54:37.219 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:35.970857+0000 mgr.smithi003.cldzke (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T05:54:37.541 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial admin user... 2023-11-15T05:54:37.542 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpr232tcro:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T05:54:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.936+0000 7fa347647700 1 Processor -- start 2023-11-15T05:54:37.946 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa347647700 1 -- start start 2023-11-15T05:54:37.946 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa347647700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:37.946 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa347647700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400fedd0 con 0x7fa340070430 2023-11-15T05:54:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa3453e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa3453e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38330/0 (socket says 172.21.15.3:38330) 2023-11-15T05:54:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa3453e3700 1 -- 172.21.15.3:0/2874679970 learned_addr learned my addr 172.21.15.3:0/2874679970 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.937+0000 7fa3453e3700 1 -- 172.21.15.3:0/2874679970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3400fef10 con 0x7fa340070430 2023-11-15T05:54:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa3453e3700 1 --2- 172.21.15.3:0/2874679970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa33c00b5f0 tx=0x7fa33c006ee0).ready entity=mon.0 client_cookie=4e16ce4c3e2ae35 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:37.948 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa32ffff700 1 -- 172.21.15.3:0/2874679970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33c00d600 con 0x7fa340070430 2023-11-15T05:54:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa32ffff700 1 -- 172.21.15.3:0/2874679970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa33c00dbd0 con 0x7fa340070430 2023-11-15T05:54:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa32ffff700 1 -- 172.21.15.3:0/2874679970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33c011c00 con 0x7fa340070430 2023-11-15T05:54:37.949 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa347647700 1 -- 172.21.15.3:0/2874679970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 msgr2=0x7fa3400fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa347647700 1 --2- 172.21.15.3:0/2874679970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa33c00b5f0 tx=0x7fa33c006ee0).stop 2023-11-15T05:54:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa347647700 1 -- 172.21.15.3:0/2874679970 shutdown_connections 2023-11-15T05:54:37.950 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa347647700 1 --2- 172.21.15.3:0/2874679970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.938+0000 7fa347647700 1 -- 172.21.15.3:0/2874679970 >> 172.21.15.3:0/2874679970 conn(0x7fa3400f63c0 msgr2=0x7fa3400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.939+0000 7fa347647700 1 -- 172.21.15.3:0/2874679970 shutdown_connections 2023-11-15T05:54:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.939+0000 7fa347647700 1 -- 172.21.15.3:0/2874679970 wait complete. 2023-11-15T05:54:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.939+0000 7fa347647700 1 Processor -- start 2023-11-15T05:54:37.952 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.939+0000 7fa347647700 1 -- start start 2023-11-15T05:54:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa347647700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa347647700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3400fa7e0 con 0x7fa340070430 2023-11-15T05:54:37.953 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa3453e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa3453e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38346/0 (socket says 172.21.15.3:38346) 2023-11-15T05:54:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa3453e3700 1 -- 172.21.15.3:0/3543823040 learned_addr learned my addr 172.21.15.3:0/3543823040 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa3453e3700 1 -- 172.21.15.3:0/3543823040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c00b200 con 0x7fa340070430 2023-11-15T05:54:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa3453e3700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa33c002f40 tx=0x7fa33c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.940+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33c011690 con 0x7fa340070430 2023-11-15T05:54:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.941+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa33c01b880 con 0x7fa340070430 2023-11-15T05:54:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.941+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3400fa980 con 0x7fa340070430 2023-11-15T05:54:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.941+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa33c01a6b0 con 0x7fa340070430 2023-11-15T05:54:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.941+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3400fad90 con 0x7fa340070430 2023-11-15T05:54:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.942+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa33c00f070 con 0x7fa340070430 2023-11-15T05:54:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.942+0000 7fa32dffb700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 0x7fa328032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.942+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa33c041540 con 0x7fa340070430 2023-11-15T05:54:37.957 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.942+0000 7fa344be2700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 0x7fa328032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.942+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa340059100 con 0x7fa340070430 2023-11-15T05:54:37.958 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.943+0000 7fa344be2700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 0x7fa328032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa3300096a0 tx=0x7fa330006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:37.959 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:37.947+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa33c016070 con 0x7fa340070430 2023-11-15T05:54:38.132 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.130+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa340045ad0 con 0x7fa32802feb0 2023-11-15T05:54:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.382+0000 7fa32dffb700 1 -- 172.21.15.3:0/3543823040 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa340045ad0 con 0x7fa32802feb0 2023-11-15T05:54:38.385 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$FxkW619/4/ON/GvDHbFLzOl8xx6gxiyGAN1NGlbBWlANbI7Q.hs5q", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700027678, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T05:54:38.387 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 msgr2=0x7fa328032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 0x7fa328032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa3300096a0 tx=0x7fa330006b40).stop 2023-11-15T05:54:38.388 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 msgr2=0x7fa3400fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa33c002f40 tx=0x7fa33c006ee0).stop 2023-11-15T05:54:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 shutdown_connections 2023-11-15T05:54:38.389 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa32802feb0 0x7fa328032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 --2- 172.21.15.3:0/3543823040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa340070430 0x7fa3400fc360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.384+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 >> 172.21.15.3:0/3543823040 conn(0x7fa3400f63c0 msgr2=0x7fa3400f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:38.390 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.386+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 shutdown_connections 2023-11-15T05:54:38.391 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:38.386+0000 7fa347647700 1 -- 172.21.15.3:0/3543823040 wait complete. 2023-11-15T05:54:38.414 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:36.880776+0000 mgr.smithi003.cldzke (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:38.414 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:36.966098+0000 mon.smithi003 (mon.0) 104 : cluster [DBG] mgrmap e16: smithi003.cldzke(active, since 2s) 2023-11-15T05:54:38.414 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:37.186855+0000 mon.smithi003 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:38.414 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:37.189499+0000 mon.smithi003 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:38.774 INFO:teuthology.orchestra.run.smithi003.stderr:Fetching dashboard port number... 2023-11-15T05:54:38.775 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmp9wfmy4xk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp3x8hjniz:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T05:54:39.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:38.132324+0000 mgr.smithi003.cldzke (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:54:39.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:38.384266+0000 mon.smithi003 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.458+0000 7f2f9e681700 1 Processor -- start 2023-11-15T05:54:39.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.458+0000 7f2f9e681700 1 -- start start 2023-11-15T05:54:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.458+0000 7f2f9e681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.458+0000 7f2f9e681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980fc5a0 con 0x7f2f980703b0 2023-11-15T05:54:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f9d67f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f9d67f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38348/0 (socket says 172.21.15.3:38348) 2023-11-15T05:54:39.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f9d67f700 1 -- 172.21.15.3:0/721614063 learned_addr learned my addr 172.21.15.3:0/721614063 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f9d67f700 1 -- 172.21.15.3:0/721614063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f981010a0 con 0x7f2f980703b0 2023-11-15T05:54:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f9d67f700 1 --2- 172.21.15.3:0/721614063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2f9400b5f0 tx=0x7f2f94006ee0).ready entity=mon.0 client_cookie=81b33d983656e685 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f8bfff700 1 -- 172.21.15.3:0/721614063 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9400d600 con 0x7f2f980703b0 2023-11-15T05:54:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f8bfff700 1 -- 172.21.15.3:0/721614063 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2f9400dbd0 con 0x7f2f980703b0 2023-11-15T05:54:39.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.459+0000 7f2f8bfff700 1 -- 172.21.15.3:0/721614063 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f94011c00 con 0x7f2f980703b0 2023-11-15T05:54:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 -- 172.21.15.3:0/721614063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 msgr2=0x7f2f98100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 --2- 172.21.15.3:0/721614063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f2f9400b5f0 tx=0x7f2f94006ee0).stop 2023-11-15T05:54:39.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 -- 172.21.15.3:0/721614063 shutdown_connections 2023-11-15T05:54:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 --2- 172.21.15.3:0/721614063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f98100bb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:39.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 -- 172.21.15.3:0/721614063 >> 172.21.15.3:0/721614063 conn(0x7f2f980f6210 msgr2=0x7f2f980f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 -- 172.21.15.3:0/721614063 shutdown_connections 2023-11-15T05:54:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.460+0000 7f2f9e681700 1 -- 172.21.15.3:0/721614063 wait complete. 2023-11-15T05:54:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9e681700 1 Processor -- start 2023-11-15T05:54:39.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9e681700 1 -- start start 2023-11-15T05:54:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9e681700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9e681700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9810cf20 con 0x7f2f980703b0 2023-11-15T05:54:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9d67f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9d67f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38354/0 (socket says 172.21.15.3:38354) 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.461+0000 7f2f9d67f700 1 -- 172.21.15.3:0/1660292864 learned_addr learned my addr 172.21.15.3:0/1660292864 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f9d67f700 1 -- 172.21.15.3:0/1660292864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9400b200 con 0x7f2f980703b0 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f9d67f700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f2f9400b190 tx=0x7f2f94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9401b410 con 0x7f2f980703b0 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9810d120 con 0x7f2f980703b0 2023-11-15T05:54:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2f9401b9e0 con 0x7f2f980703b0 2023-11-15T05:54:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9401a880 con 0x7f2f980703b0 2023-11-15T05:54:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.462+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9810d530 con 0x7f2f980703b0 2023-11-15T05:54:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.463+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f2f9400f070 con 0x7f2f980703b0 2023-11-15T05:54:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.463+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98045ad0 con 0x7f2f980703b0 2023-11-15T05:54:39.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.463+0000 7f2f89ffb700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 0x7f2f80032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.463+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f94041500 con 0x7f2f980703b0 2023-11-15T05:54:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.463+0000 7f2f9ce7e700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 0x7f2f80032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.464+0000 7f2f9ce7e700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 0x7f2f80032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2f8c0096a0 tx=0x7f2f8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:39.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.468+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f94019cd0 con 0x7f2f980703b0 2023-11-15T05:54:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.632+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2f98101c00 con 0x7f2f980703b0 2023-11-15T05:54:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.633+0000 7f2f89ffb700 1 -- 172.21.15.3:0/1660292864 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2f94016070 con 0x7f2f980703b0 2023-11-15T05:54:39.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 8443 2023-11-15T05:54:39.637 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 msgr2=0x7f2f80032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 0x7f2f80032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2f8c0096a0 tx=0x7f2f8c006b40).stop 2023-11-15T05:54:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 msgr2=0x7f2f9810ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f2f9400b190 tx=0x7f2f94006ee0).stop 2023-11-15T05:54:39.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 shutdown_connections 2023-11-15T05:54:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2f8002feb0 0x7f2f80032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 --2- 172.21.15.3:0/1660292864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f980703b0 0x7f2f9810ca30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 >> 172.21.15.3:0/1660292864 conn(0x7f2f980f6210 msgr2=0x7f2f980f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 shutdown_connections 2023-11-15T05:54:39.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-15T05:54:39.635+0000 7f2f9e681700 1 -- 172.21.15.3:0/1660292864 wait complete. 2023-11-15T05:54:40.007 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T05:54:40.019 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-15T05:54:40.019 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T05:54:40.028 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-15T05:54:40.029 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-15T05:54:40.029 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T05:54:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:Ceph Dashboard is now available at: 2023-11-15T05:54:40.034 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.034 INFO:teuthology.orchestra.run.smithi003.stderr: URL: https://smithi003.front.sepia.ceph.com:8443/ 2023-11-15T05:54:40.034 INFO:teuthology.orchestra.run.smithi003.stderr: User: admin 2023-11-15T05:54:40.034 INFO:teuthology.orchestra.run.smithi003.stderr: Password: idtnnmxutt 2023-11-15T05:54:40.034 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.035 INFO:teuthology.orchestra.run.smithi003.stderr:You can access the Ceph CLI with: 2023-11-15T05:54:40.035 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.035 INFO:teuthology.orchestra.run.smithi003.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T05:54:40.035 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.036 INFO:teuthology.orchestra.run.smithi003.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T05:54:40.036 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.036 INFO:teuthology.orchestra.run.smithi003.stderr: ceph telemetry on 2023-11-15T05:54:40.036 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.036 INFO:teuthology.orchestra.run.smithi003.stderr:For more information see: 2023-11-15T05:54:40.037 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.037 INFO:teuthology.orchestra.run.smithi003.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T05:54:40.037 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T05:54:40.038 INFO:teuthology.orchestra.run.smithi003.stderr:Bootstrap complete. 2023-11-15T05:54:40.038 INFO:teuthology.orchestra.run.smithi003.stderr:Releasing lock 140643497687528 on /run/cephadm/412cc4c0-837b-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:54:40.038 INFO:teuthology.orchestra.run.smithi003.stderr:Lock 140643497687528 released on /run/cephadm/412cc4c0-837b-11ee-8dbe-212e2dc638e7.lock 2023-11-15T05:54:40.074 INFO:tasks.cephadm:Fetching config... 2023-11-15T05:54:40.075 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:54:40.075 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T05:54:40.097 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T05:54:40.098 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:54:40.098 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T05:54:40.158 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T05:54:40.159 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:54:40.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/mon.smithi003/keyring of=/dev/stdout 2023-11-15T05:54:40.225 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:39.634778+0000 mon.smithi003 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.3:0/1660292864' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T05:54:40.229 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T05:54:40.229 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-15T05:54:40.229 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T05:54:40.289 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T05:54:40.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxPBVAYPM6VLE1GY1E3IaalzbSwAvuTKN8Rh93YagBfBwfrdkhW3rTAu8BaoHUE0f9pbp6NksrB6miyKZld9+iGvRdZ5mLHR1ZHMDddZRfT2AoWqtOAoxqvyS6r5WbwRUtOC8PmZznNjpqZ+3Wk2dIeFTGomRsLIzyvmeBdHmRnyMX+xGWm3J5G0kA3ZeZoltsnySIy4e7YszPdZ2N+WNU/n/ZxVB0q7qHuSGo9t4oIhJJSblbL3pitRJAmwG9IjUzr1byZziakqlGqYtRJCqD9BSBEsY5l+wUTyF4llgaOX+qd9hVl7rMjil8OCKgDiCqPZ9Wzw5QdrWOIY7ZcsjukQEG7+vLd7mIdBRugKGcbYAVmYcj1HVz59IBVjtHlRD7QrLCAPH7QbXn3UrB+E3BV/wbUThkIr8g9F7RoIZ/zdX+2sxr6QpYXhEFG7A0LEykA7eEz4i/kPMPRdmwAz/kLRyg0rWsbh8rqCJgUfyJybXot5BOtSDbJqicT93uB3U= ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T05:54:40.374 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxPBVAYPM6VLE1GY1E3IaalzbSwAvuTKN8Rh93YagBfBwfrdkhW3rTAu8BaoHUE0f9pbp6NksrB6miyKZld9+iGvRdZ5mLHR1ZHMDddZRfT2AoWqtOAoxqvyS6r5WbwRUtOC8PmZznNjpqZ+3Wk2dIeFTGomRsLIzyvmeBdHmRnyMX+xGWm3J5G0kA3ZeZoltsnySIy4e7YszPdZ2N+WNU/n/ZxVB0q7qHuSGo9t4oIhJJSblbL3pitRJAmwG9IjUzr1byZziakqlGqYtRJCqD9BSBEsY5l+wUTyF4llgaOX+qd9hVl7rMjil8OCKgDiCqPZ9Wzw5QdrWOIY7ZcsjukQEG7+vLd7mIdBRugKGcbYAVmYcj1HVz59IBVjtHlRD7QrLCAPH7QbXn3UrB+E3BV/wbUThkIr8g9F7RoIZ/zdX+2sxr6QpYXhEFG7A0LEykA7eEz4i/kPMPRdmwAz/kLRyg0rWsbh8rqCJgUfyJybXot5BOtSDbJqicT93uB3U= ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:54:40.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxPBVAYPM6VLE1GY1E3IaalzbSwAvuTKN8Rh93YagBfBwfrdkhW3rTAu8BaoHUE0f9pbp6NksrB6miyKZld9+iGvRdZ5mLHR1ZHMDddZRfT2AoWqtOAoxqvyS6r5WbwRUtOC8PmZznNjpqZ+3Wk2dIeFTGomRsLIzyvmeBdHmRnyMX+xGWm3J5G0kA3ZeZoltsnySIy4e7YszPdZ2N+WNU/n/ZxVB0q7qHuSGo9t4oIhJJSblbL3pitRJAmwG9IjUzr1byZziakqlGqYtRJCqD9BSBEsY5l+wUTyF4llgaOX+qd9hVl7rMjil8OCKgDiCqPZ9Wzw5QdrWOIY7ZcsjukQEG7+vLd7mIdBRugKGcbYAVmYcj1HVz59IBVjtHlRD7QrLCAPH7QbXn3UrB+E3BV/wbUThkIr8g9F7RoIZ/zdX+2sxr6QpYXhEFG7A0LEykA7eEz4i/kPMPRdmwAz/kLRyg0rWsbh8rqCJgUfyJybXot5BOtSDbJqicT93uB3U= ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T05:54:40.431 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxPBVAYPM6VLE1GY1E3IaalzbSwAvuTKN8Rh93YagBfBwfrdkhW3rTAu8BaoHUE0f9pbp6NksrB6miyKZld9+iGvRdZ5mLHR1ZHMDddZRfT2AoWqtOAoxqvyS6r5WbwRUtOC8PmZznNjpqZ+3Wk2dIeFTGomRsLIzyvmeBdHmRnyMX+xGWm3J5G0kA3ZeZoltsnySIy4e7YszPdZ2N+WNU/n/ZxVB0q7qHuSGo9t4oIhJJSblbL3pitRJAmwG9IjUzr1byZziakqlGqYtRJCqD9BSBEsY5l+wUTyF4llgaOX+qd9hVl7rMjil8OCKgDiCqPZ9Wzw5QdrWOIY7ZcsjukQEG7+vLd7mIdBRugKGcbYAVmYcj1HVz59IBVjtHlRD7QrLCAPH7QbXn3UrB+E3BV/wbUThkIr8g9F7RoIZ/zdX+2sxr6QpYXhEFG7A0LEykA7eEz4i/kPMPRdmwAz/kLRyg0rWsbh8rqCJgUfyJybXot5BOtSDbJqicT93uB3U= ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7 2023-11-15T05:54:40.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T05:54:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 -- 172.21.15.3:0/2198234846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 msgr2=0x7f52ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 --2- 172.21.15.3:0/2198234846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec0f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f52dc009960 tx=0x7f52dc004d10).stop 2023-11-15T05:54:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 -- 172.21.15.3:0/2198234846 shutdown_connections 2023-11-15T05:54:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 --2- 172.21.15.3:0/2198234846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec0f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 -- 172.21.15.3:0/2198234846 >> 172.21.15.3:0/2198234846 conn(0x7f52ec0f1c70 msgr2=0x7f52ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.678+0000 7f52f1a89700 1 -- 172.21.15.3:0/2198234846 shutdown_connections 2023-11-15T05:54:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.679+0000 7f52f1a89700 1 -- 172.21.15.3:0/2198234846 wait complete. 2023-11-15T05:54:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.679+0000 7f52f1a89700 1 Processor -- start 2023-11-15T05:54:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.679+0000 7f52f1a89700 1 -- start start 2023-11-15T05:54:41.684 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52f1a89700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52f1a89700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec106ba0 con 0x7f52ec0f6510 2023-11-15T05:54:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52eb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38370/0 (socket says 172.21.15.3:38370) 2023-11-15T05:54:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52eb7fe700 1 -- 172.21.15.3:0/3828838789 learned_addr learned my addr 172.21.15.3:0/3828838789 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:54:41.685 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52eb7fe700 1 -- 172.21.15.3:0/3828838789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc009640 con 0x7f52ec0f6510 2023-11-15T05:54:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52eb7fe700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f52dc004a10 tx=0x7f52dc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52dc019070 con 0x7f52ec0f6510 2023-11-15T05:54:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f52dc00beb0 con 0x7f52ec0f6510 2023-11-15T05:54:41.686 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec106da0 con 0x7f52ec0f6510 2023-11-15T05:54:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52dc01e420 con 0x7f52ec0f6510 2023-11-15T05:54:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.680+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec107230 con 0x7f52ec0f6510 2023-11-15T05:54:41.687 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.681+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52dc025060 con 0x7f52ec0f6510 2023-11-15T05:54:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.681+0000 7f52e3fff700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 0x7f52d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:54:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.681+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52dc041000 con 0x7f52ec0f6510 2023-11-15T05:54:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.681+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec045ad0 con 0x7f52ec0f6510 2023-11-15T05:54:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.682+0000 7f52eaffd700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 0x7f52d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:54:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.682+0000 7f52eaffd700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 0x7f52d8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f52d40096a0 tx=0x7f52d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:54:41.689 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.686+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52dc028440 con 0x7f52ec0f6510 2023-11-15T05:54:41.854 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.853+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f52ec0fbf50 con 0x7f52ec0f6510 2023-11-15T05:54:41.858 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.857+0000 7f52e3fff700 1 -- 172.21.15.3:0/3828838789 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f52ec0fbf50 con 0x7f52ec0f6510 2023-11-15T05:54:41.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 msgr2=0x7f52d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:41.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 0x7f52d8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f52d40096a0 tx=0x7f52d4006b40).stop 2023-11-15T05:54:41.861 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 msgr2=0x7f52ec1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:54:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f52dc004a10 tx=0x7f52dc004d10).stop 2023-11-15T05:54:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 shutdown_connections 2023-11-15T05:54:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f52d802ffb0 0x7f52d8032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 --2- 172.21.15.3:0/3828838789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f52ec0f6510 0x7f52ec1066b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:54:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 >> 172.21.15.3:0/3828838789 conn(0x7f52ec0f1c70 msgr2=0x7f52ec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:54:41.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 shutdown_connections 2023-11-15T05:54:41.863 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:54:41.859+0000 7f52f1a89700 1 -- 172.21.15.3:0/3828838789 wait complete. 2023-11-15T05:54:42.235 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi137 2023-11-15T05:54:42.235 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:54:42.235 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.conf 2023-11-15T05:54:42.257 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2023-11-15T05:54:42.257 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T05:54:42.317 INFO:tasks.cephadm:Adding host smithi137 to orchestrator... 2023-11-15T05:54:42.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi137 2023-11-15T05:54:42.761 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:41.491646+0000 mon.smithi003 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi003.cldzke(active, since 6s) 2023-11-15T05:54:42.761 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:41.858733+0000 mon.smithi003 (mon.0) 110 : audit [INF] from='client.? 172.21.15.3:0/3828838789' entity='client.admin' 2023-11-15T05:54:44.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:43.081253+0000 mon.smithi003 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:44.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:43.327467+0000 mon.smithi003 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:44.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:44.010767+0000 mon.smithi003 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:46.521213+0000 mon.smithi003 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:46.837752+0000 mon.smithi003 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:54:46.843043+0000 mon.smithi003 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:54:47.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:54:46.850112+0000 mgr.smithi003.cldzke (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi003 on smithi003 2023-11-15T05:54:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:55 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:54.964794+0000 mgr.smithi003.cldzke (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:54:58.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:57 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:56.965215+0000 mgr.smithi003.cldzke (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:54:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:54:58.965575+0000 mgr.smithi003.cldzke (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:01.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:00.965814+0000 mgr.smithi003.cldzke (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:02.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 -- 172.21.15.137:0/495050589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 msgr2=0x7f38e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:02.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 --2- 172.21.15.137:0/495050589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40f55c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f38d8009fb0 tx=0x7f38d8009b30).stop 2023-11-15T05:55:02.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 -- 172.21.15.137:0/495050589 shutdown_connections 2023-11-15T05:55:02.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 --2- 172.21.15.137:0/495050589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40f55c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:02.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 -- 172.21.15.137:0/495050589 >> 172.21.15.137:0/495050589 conn(0x7f38e40f09a0 msgr2=0x7f38e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:02.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 -- 172.21.15.137:0/495050589 shutdown_connections 2023-11-15T05:55:02.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 -- 172.21.15.137:0/495050589 wait complete. 2023-11-15T05:55:02.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.811+0000 7f38e359e700 1 Processor -- start 2023-11-15T05:55:02.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e359e700 1 -- start start 2023-11-15T05:55:02.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40fb5a0 con 0x7f38e40f51a0 2023-11-15T05:55:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33720/0 (socket says 172.21.15.137:33720) 2023-11-15T05:55:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e259c700 1 -- 172.21.15.137:0/2680344680 learned_addr learned my addr 172.21.15.137:0/2680344680 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e259c700 1 -- 172.21.15.137:0/2680344680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d8005040 con 0x7f38e40f51a0 2023-11-15T05:55:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.812+0000 7f38e259c700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f38d8013510 tx=0x7f38d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d8009320 con 0x7f38e40f51a0 2023-11-15T05:55:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e40fb740 con 0x7f38e40f51a0 2023-11-15T05:55:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38d8013f00 con 0x7f38e40f51a0 2023-11-15T05:55:02.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d8014dc0 con 0x7f38e40f51a0 2023-11-15T05:55:02.822 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e40fbbd0 con 0x7f38e40f51a0 2023-11-15T05:55:02.825 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.813+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38d8014420 con 0x7f38e40f51a0 2023-11-15T05:55:02.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.814+0000 7f38d2ffd700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 0x7f38cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:02.826 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.814+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38d8041050 con 0x7f38e40f51a0 2023-11-15T05:55:02.831 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.814+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e4059100 con 0x7f38e40f51a0 2023-11-15T05:55:02.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.814+0000 7f38e1d9b700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 0x7f38cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:02.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.814+0000 7f38e1d9b700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 0x7f38cc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f38d40096a0 tx=0x7f38d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:02.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.819+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38d80095f0 con 0x7f38e40f51a0 2023-11-15T05:55:02.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:02.989+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f38e40fc340 con 0x7f38cc02ff60 2023-11-15T05:55:03.587 INFO:teuthology.orchestra.run.smithi137.stdout:Added host 'smithi137' 2023-11-15T05:55:03.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.584+0000 7f38d2ffd700 1 -- 172.21.15.137:0/2680344680 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f38e40fc340 con 0x7f38cc02ff60 2023-11-15T05:55:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.586+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 msgr2=0x7f38cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.586+0000 7f38e359e700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 0x7f38cc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f38d40096a0 tx=0x7f38d4006b40).stop 2023-11-15T05:55:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 msgr2=0x7f38e40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f38d8013510 tx=0x7f38d800b040).stop 2023-11-15T05:55:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 shutdown_connections 2023-11-15T05:55:03.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f38cc02ff60 0x7f38cc032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:03.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 --2- 172.21.15.137:0/2680344680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f38e40f51a0 0x7f38e40fd120 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:03.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 >> 172.21.15.137:0/2680344680 conn(0x7f38e40f09a0 msgr2=0x7f38e405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:03.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 shutdown_connections 2023-11-15T05:55:03.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:03.587+0000 7f38e359e700 1 -- 172.21.15.137:0/2680344680 wait complete. 2023-11-15T05:55:03.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T05:55:04.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:02.966175+0000 mgr.smithi003.cldzke (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:04.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:02.991803+0000 mgr.smithi003.cldzke (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:55:04.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:03.584915+0000 mon.smithi003 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:55:05.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 -- 172.21.15.137:0/2877888523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 msgr2=0x7f64c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:05.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 --2- 172.21.15.137:0/2877888523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c00f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f64b4009fb0 tx=0x7f64b4009b30).stop 2023-11-15T05:55:05.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 -- 172.21.15.137:0/2877888523 shutdown_connections 2023-11-15T05:55:05.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 --2- 172.21.15.137:0/2877888523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c00f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 -- 172.21.15.137:0/2877888523 >> 172.21.15.137:0/2877888523 conn(0x7f64c00f0b70 msgr2=0x7f64c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 -- 172.21.15.137:0/2877888523 shutdown_connections 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.065+0000 7f64c63d5700 1 -- 172.21.15.137:0/2877888523 wait complete. 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64c63d5700 1 Processor -- start 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64c63d5700 1 -- start start 2023-11-15T05:55:05.069 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64c63d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:05.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64c63d5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0060220 con 0x7f64c00f5410 2023-11-15T05:55:05.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64bffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:05.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64bffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33736/0 (socket says 172.21.15.137:33736) 2023-11-15T05:55:05.070 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64bffff700 1 -- 172.21.15.137:0/181094692 learned_addr learned my addr 172.21.15.137:0/181094692 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:05.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.066+0000 7f64bffff700 1 -- 172.21.15.137:0/181094692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b4005040 con 0x7f64c00f5410 2023-11-15T05:55:05.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64bffff700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f64b4013510 tx=0x7f64b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:05.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b4009320 con 0x7f64c00f5410 2023-11-15T05:55:05.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0060420 con 0x7f64c00f5410 2023-11-15T05:55:05.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64b4013f00 con 0x7f64c00f5410 2023-11-15T05:55:05.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b4014d90 con 0x7f64c00f5410 2023-11-15T05:55:05.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.067+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c0109b00 con 0x7f64c00f5410 2023-11-15T05:55:05.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64b4014420 con 0x7f64c00f5410 2023-11-15T05:55:05.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c0059100 con 0x7f64c00f5410 2023-11-15T05:55:05.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64bcff9700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:05.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64b40410c0 con 0x7f64c00f5410 2023-11-15T05:55:05.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64bf7fe700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:05.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.068+0000 7f64bf7fe700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 0x7f64a8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64b00096a0 tx=0x7f64b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:05.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.072+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64b40095f0 con 0x7f64c00f5410 2023-11-15T05:55:05.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.242+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64c00f1d70 con 0x7f64a802ff60 2023-11-15T05:55:05.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.244+0000 7f64bcff9700 1 -- 172.21.15.137:0/181094692 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+155 (secure 0 0 0) 0x7f64c00f1d70 con 0x7f64a802ff60 2023-11-15T05:55:05.246 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:05.246 INFO:teuthology.orchestra.run.smithi137.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "smithi137", "hostname": "smithi137", "labels": [], "status": ""}] 2023-11-15T05:55:05.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 msgr2=0x7f64a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:05.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 0x7f64a8032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64b00096a0 tx=0x7f64b0006b40).stop 2023-11-15T05:55:05.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 msgr2=0x7f64c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:05.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f64b4013510 tx=0x7f64b400b040).stop 2023-11-15T05:55:05.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 shutdown_connections 2023-11-15T05:55:05.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:05.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 --2- 172.21.15.137:0/181094692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64c00f5410 0x7f64c0109450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:05.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 >> 172.21.15.137:0/181094692 conn(0x7f64c00f0b70 msgr2=0x7f64c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:05.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.245+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 shutdown_connections 2023-11-15T05:55:05.249 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:05.246+0000 7f64c63d5700 1 -- 172.21.15.137:0/181094692 wait complete. 2023-11-15T05:55:05.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:55:03.585368+0000 mgr.smithi003.cldzke (mgr.14164) 12 : cephadm [INF] Added host smithi137 2023-11-15T05:55:05.624 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T05:55:05.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T05:55:06.369 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:04.966473+0000 mgr.smithi003.cldzke (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:06.369 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:05.244872+0000 mgr.smithi003.cldzke (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T05:55:06.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 -- 172.21.15.3:0/3699058311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 msgr2=0x7f66e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:06.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/3699058311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e40f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f66d4009fb0 tx=0x7f66d4009b30).stop 2023-11-15T05:55:06.875 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 -- 172.21.15.3:0/3699058311 shutdown_connections 2023-11-15T05:55:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/3699058311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e40f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 -- 172.21.15.3:0/3699058311 >> 172.21.15.3:0/3699058311 conn(0x7f66e40f1c70 msgr2=0x7f66e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 -- 172.21.15.3:0/3699058311 shutdown_connections 2023-11-15T05:55:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 -- 172.21.15.3:0/3699058311 wait complete. 2023-11-15T05:55:06.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.874+0000 7f66ea1b5700 1 Processor -- start 2023-11-15T05:55:06.878 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66ea1b5700 1 -- start start 2023-11-15T05:55:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66ea1b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66ea1b5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4069e20 con 0x7f66e40f6510 2023-11-15T05:55:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66e37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33670/0 (socket says 172.21.15.3:33670) 2023-11-15T05:55:06.879 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66e37fe700 1 -- 172.21.15.3:0/1743080851 learned_addr learned my addr 172.21.15.3:0/1743080851 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-15T05:55:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66e37fe700 1 -- 172.21.15.3:0/1743080851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4005040 con 0x7f66e40f6510 2023-11-15T05:55:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66e37fe700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f66d4013510 tx=0x7f66d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.875+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66d4009320 con 0x7f66e40f6510 2023-11-15T05:55:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.876+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66d4013f00 con 0x7f66e40f6510 2023-11-15T05:55:06.880 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.876+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e4069fc0 con 0x7f66e40f6510 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.876+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66d4014d90 con 0x7f66e40f6510 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.876+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e406a450 con 0x7f66e40f6510 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.876+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66d40263a0 con 0x7f66e40f6510 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.877+0000 7f66cbfff700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 0x7f66cc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.877+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e4059100 con 0x7f66e40f6510 2023-11-15T05:55:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.877+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66d40412b0 con 0x7f66e40f6510 2023-11-15T05:55:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.877+0000 7f66e2ffd700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 0x7f66cc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.877+0000 7f66e2ffd700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 0x7f66cc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f66d80096a0 tx=0x7f66d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:06.881+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66d4012920 con 0x7f66e40f6510 2023-11-15T05:55:07.058 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:07.057+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f66e40fbf60 con 0x7f66e40f6510 2023-11-15T05:55:08.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.037+0000 7f66cbfff700 1 -- 172.21.15.3:0/1743080851 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f66d4012330 con 0x7f66e40f6510 2023-11-15T05:55:08.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 msgr2=0x7f66cc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:08.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 0x7f66cc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f66d80096a0 tx=0x7f66d8006b40).stop 2023-11-15T05:55:08.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 msgr2=0x7f66e406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:08.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f66d4013510 tx=0x7f66d400b040).stop 2023-11-15T05:55:08.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 shutdown_connections 2023-11-15T05:55:08.042 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f66cc02ffb0 0x7f66cc032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:08.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 --2- 172.21.15.3:0/1743080851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66e40f6510 0x7f66e406b9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:08.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 >> 172.21.15.3:0/1743080851 conn(0x7f66e40f1c70 msgr2=0x7f66e40f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:08.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 shutdown_connections 2023-11-15T05:55:08.043 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-15T05:55:08.040+0000 7f66ea1b5700 1 -- 172.21.15.3:0/1743080851 wait complete. 2023-11-15T05:55:08.044 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-11-15T05:55:08.331 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:06.966736+0000 mgr.smithi003.cldzke (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:08.333 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:07.058180+0000 mon.smithi003 (mon.0) 118 : audit [INF] from='client.? 172.21.15.3:0/1743080851' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T05:55:08.512 INFO:tasks.cephadm:Adding mon.smithi003 on smithi003 2023-11-15T05:55:08.513 INFO:tasks.cephadm:Adding mon.smithi137 on smithi137 2023-11-15T05:55:08.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi003:172.21.15.3=smithi003;smithi137:172.21.15.137=smithi137' 2023-11-15T05:55:09.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:08.038337+0000 mon.smithi003 (mon.0) 119 : audit [INF] from='client.? 172.21.15.3:0/1743080851' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T05:55:09.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:08.038428+0000 mon.smithi003 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T05:55:09.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 -- 172.21.15.137:0/3450547674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 msgr2=0x7f64f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:09.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 --2- 172.21.15.137:0/3450547674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f40f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f64e8009fb0 tx=0x7f64e8009b30).stop 2023-11-15T05:55:09.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 -- 172.21.15.137:0/3450547674 shutdown_connections 2023-11-15T05:55:09.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 --2- 172.21.15.137:0/3450547674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f40f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:09.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 -- 172.21.15.137:0/3450547674 >> 172.21.15.137:0/3450547674 conn(0x7f64f40f0b70 msgr2=0x7f64f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:09.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 -- 172.21.15.137:0/3450547674 shutdown_connections 2023-11-15T05:55:09.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.705+0000 7f64f8c73700 1 -- 172.21.15.137:0/3450547674 wait complete. 2023-11-15T05:55:09.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f8c73700 1 Processor -- start 2023-11-15T05:55:09.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f8c73700 1 -- start start 2023-11-15T05:55:09.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f8c73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:09.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f8c73700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f4060220 con 0x7f64f40f5410 2023-11-15T05:55:09.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:09.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55168/0 (socket says 172.21.15.137:55168) 2023-11-15T05:55:09.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f259c700 1 -- 172.21.15.137:0/2376396008 learned_addr learned my addr 172.21.15.137:0/2376396008 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:09.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.706+0000 7f64f259c700 1 -- 172.21.15.137:0/2376396008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8005040 con 0x7f64f40f5410 2023-11-15T05:55:09.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64f259c700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f64e8014000 tx=0x7f64e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:09.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e80095f0 con 0x7f64f40f5410 2023-11-15T05:55:09.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f4060420 con 0x7f64f40f5410 2023-11-15T05:55:09.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e8007e70 con 0x7f64f40f5410 2023-11-15T05:55:09.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e8014ea0 con 0x7f64f40f5410 2023-11-15T05:55:09.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.707+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f4109b00 con 0x7f64f40f5410 2023-11-15T05:55:09.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64e8025070 con 0x7f64f40f5410 2023-11-15T05:55:09.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f4059100 con 0x7f64f40f5410 2023-11-15T05:55:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64e2ffd700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64e80409e0 con 0x7f64f40f5410 2023-11-15T05:55:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64f1d9b700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.708+0000 7f64f1d9b700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 0x7f64dc032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f64e40096a0 tx=0x7f64e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:09.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.712+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64e800f4e0 con 0x7f64f40f5410 2023-11-15T05:55:09.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.888+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 --> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi003:172.21.15.3=smithi003;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f64f40f1e20 con 0x7f64dc02ff60 2023-11-15T05:55:09.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.894+0000 7f64e2ffd700 1 -- 172.21.15.137:0/2376396008 <== mgr.14164 v2:172.21.15.3:6800/4263843884 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f64f40f1e20 con 0x7f64dc02ff60 2023-11-15T05:55:09.895 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mon update... 2023-11-15T05:55:09.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 msgr2=0x7f64dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:09.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 0x7f64dc032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f64e40096a0 tx=0x7f64e4006b40).stop 2023-11-15T05:55:09.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 msgr2=0x7f64f4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:09.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f64e8014000 tx=0x7f64e800b040).stop 2023-11-15T05:55:09.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 shutdown_connections 2023-11-15T05:55:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 --2- 172.21.15.137:0/2376396008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64f40f5410 0x7f64f4109450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 >> 172.21.15.137:0/2376396008 conn(0x7f64f40f0b70 msgr2=0x7f64f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 shutdown_connections 2023-11-15T05:55:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:09.895+0000 7f64f8c73700 1 -- 172.21.15.137:0/2376396008 wait complete. 2023-11-15T05:55:10.233 DEBUG:teuthology.orchestra.run.smithi137:mon.smithi137> sudo journalctl -f -n 0 -u ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137.service 2023-11-15T05:55:10.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:10.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:10.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:-- Logs begin at Wed 2023-11-15 05:39:50 UTC. -- 2023-11-15T05:55:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:08.967056+0000 mgr.smithi003.cldzke (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:09.894727+0000 mon.smithi003 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:55:11.426 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 -- 172.21.15.137:0/2010769102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 msgr2=0x7fa3200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:11.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 --2- 172.21.15.137:0/2010769102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa3200f6930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa310009960 tx=0x7fa310004d10).stop 2023-11-15T05:55:11.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 -- 172.21.15.137:0/2010769102 shutdown_connections 2023-11-15T05:55:11.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 --2- 172.21.15.137:0/2010769102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa3200f6930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:11.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 -- 172.21.15.137:0/2010769102 >> 172.21.15.137:0/2010769102 conn(0x7fa3200f1c70 msgr2=0x7fa3200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:11.427 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 -- 172.21.15.137:0/2010769102 shutdown_connections 2023-11-15T05:55:11.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.423+0000 7fa3284bb700 1 -- 172.21.15.137:0/2010769102 wait complete. 2023-11-15T05:55:11.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa3284bb700 1 Processor -- start 2023-11-15T05:55:11.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa3284bb700 1 -- start start 2023-11-15T05:55:11.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa3284bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:11.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa3284bb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa320106c20 con 0x7fa3200f6510 2023-11-15T05:55:11.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa326257700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:11.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa326257700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55198/0 (socket says 172.21.15.137:55198) 2023-11-15T05:55:11.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa326257700 1 -- 172.21.15.137:0/4004775883 learned_addr learned my addr 172.21.15.137:0/4004775883 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:11.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa326257700 1 -- 172.21.15.137:0/4004775883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310009640 con 0x7fa3200f6510 2023-11-15T05:55:11.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.424+0000 7fa326257700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa310004b70 tx=0x7fa310004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:11.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.425+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa310018070 con 0x7fa3200f6510 2023-11-15T05:55:11.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.425+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa31000b870 con 0x7fa3200f6510 2023-11-15T05:55:11.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.425+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa320106e20 con 0x7fa3200f6510 2023-11-15T05:55:11.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.425+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa31001c560 con 0x7fa3200f6510 2023-11-15T05:55:11.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.425+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3201072b0 con 0x7fa3200f6510 2023-11-15T05:55:11.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:09.890407+0000 mgr.smithi003.cldzke (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi003:172.21.15.3=smithi003;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T05:55:11.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:55:09.891918+0000 mgr.smithi003.cldzke (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi003:172.21.15.3=smithi003;smithi137:172.21.15.137=smithi137;count:2 2023-11-15T05:55:11.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa310013630 con 0x7fa3200f6510 2023-11-15T05:55:11.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa320045ad0 con 0x7fa3200f6510 2023-11-15T05:55:11.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa31effd700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 0x7fa308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:11.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3100411d0 con 0x7fa3200f6510 2023-11-15T05:55:11.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa325a56700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 0x7fa308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:11.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.426+0000 7fa325a56700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 0x7fa308032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa3140096a0 tx=0x7fa314006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:11.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.430+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa310004810 con 0x7fa3200f6510 2023-11-15T05:55:11.637 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.636+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3200fbf60 con 0x7fa3200f6510 2023-11-15T05:55:11.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.637+0000 7fa31effd700 1 -- 172.21.15.137:0/4004775883 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa310012380 con 0x7fa3200f6510 2023-11-15T05:55:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:11.639 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:11.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 msgr2=0x7fa308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:11.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 0x7fa308032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa3140096a0 tx=0x7fa314006b40).stop 2023-11-15T05:55:11.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 msgr2=0x7fa320106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:11.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa310004b70 tx=0x7fa310004d10).stop 2023-11-15T05:55:11.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 shutdown_connections 2023-11-15T05:55:11.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa30802ff60 0x7fa308032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:11.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 --2- 172.21.15.137:0/4004775883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa3200f6510 0x7fa320106730 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:11.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 >> 172.21.15.137:0/4004775883 conn(0x7fa3200f1c70 msgr2=0x7fa3200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:11.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 shutdown_connections 2023-11-15T05:55:11.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:11.638+0000 7fa3284bb700 1 -- 172.21.15.137:0/4004775883 wait complete. 2023-11-15T05:55:11.642 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:12.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:10.967310+0000 mgr.smithi003.cldzke (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:12.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:11.638059+0000 mon.smithi003 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.137:0/4004775883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:13.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:13.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:14.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- 172.21.15.137:0/29955249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 msgr2=0x7f6bf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:14.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 --2- 172.21.15.137:0/29955249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf40f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6be8009fb0 tx=0x7f6be8009b30).stop 2023-11-15T05:55:14.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- 172.21.15.137:0/29955249 shutdown_connections 2023-11-15T05:55:14.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 --2- 172.21.15.137:0/29955249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf40f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:14.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- 172.21.15.137:0/29955249 >> 172.21.15.137:0/29955249 conn(0x7f6bf40f1c70 msgr2=0x7f6bf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:14.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- 172.21.15.137:0/29955249 shutdown_connections 2023-11-15T05:55:14.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- 172.21.15.137:0/29955249 wait complete. 2023-11-15T05:55:14.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 Processor -- start 2023-11-15T05:55:14.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.410+0000 7f6bfac32700 1 -- start start 2023-11-15T05:55:14.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bfac32700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:14.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bfac32700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf4106c20 con 0x7f6bf40f6510 2023-11-15T05:55:14.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bf89ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:14.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bf89ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55206/0 (socket says 172.21.15.137:55206) 2023-11-15T05:55:14.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bf89ce700 1 -- 172.21.15.137:0/3852694839 learned_addr learned my addr 172.21.15.137:0/3852694839 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:14.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bf89ce700 1 -- 172.21.15.137:0/3852694839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8005040 con 0x7f6bf40f6510 2023-11-15T05:55:14.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.411+0000 7f6bf89ce700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6be8013510 tx=0x7f6be800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:14.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be8009320 con 0x7f6bf40f6510 2023-11-15T05:55:14.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf4106e20 con 0x7f6bf40f6510 2023-11-15T05:55:14.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6be8013f00 con 0x7f6bf40f6510 2023-11-15T05:55:14.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be8014d90 con 0x7f6bf40f6510 2023-11-15T05:55:14.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf41072b0 con 0x7f6bf40f6510 2023-11-15T05:55:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6be8014420 con 0x7f6bf40f6510 2023-11-15T05:55:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.412+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4045ad0 con 0x7f6bf40f6510 2023-11-15T05:55:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.413+0000 7f6bf17fa700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 0x7f6be4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.413+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6be8041000 con 0x7f6bf40f6510 2023-11-15T05:55:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.413+0000 7f6bf3fff700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 0x7f6be4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:14.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.413+0000 7f6bf3fff700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 0x7f6be4032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6be00096a0 tx=0x7f6be0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:14.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.417+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6be80095f0 con 0x7f6bf40f6510 2023-11-15T05:55:14.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:12.967669+0000 mgr.smithi003.cldzke (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:14.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.622+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bf40fbf60 con 0x7f6bf40f6510 2023-11-15T05:55:14.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.622+0000 7f6bf17fa700 1 -- 172.21.15.137:0/3852694839 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6be801db70 con 0x7f6bf40f6510 2023-11-15T05:55:14.624 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:14.625 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:14.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 msgr2=0x7f6be4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:14.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 0x7f6be4032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f6be00096a0 tx=0x7f6be0006b40).stop 2023-11-15T05:55:14.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 msgr2=0x7f6bf4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:14.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6be8013510 tx=0x7f6be800b040).stop 2023-11-15T05:55:14.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 shutdown_connections 2023-11-15T05:55:14.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6be402ff60 0x7f6be4032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:14.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 --2- 172.21.15.137:0/3852694839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6bf40f6510 0x7f6bf4106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:14.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 >> 172.21.15.137:0/3852694839 conn(0x7f6bf40f1c70 msgr2=0x7f6bf40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:14.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 shutdown_connections 2023-11-15T05:55:14.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:14.624+0000 7f6bfac32700 1 -- 172.21.15.137:0/3852694839 wait complete. 2023-11-15T05:55:14.628 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:15.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:14.624018+0000 mon.smithi003 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.137:0/3852694839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:16.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:16.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:16.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:14.968024+0000 mgr.smithi003.cldzke (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:17.138 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 -- 172.21.15.137:0/1179238506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 msgr2=0x7f7b600f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:17.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 --2- 172.21.15.137:0/1179238506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b600f4be0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7b54009fb0 tx=0x7f7b54009b30).stop 2023-11-15T05:55:17.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 -- 172.21.15.137:0/1179238506 shutdown_connections 2023-11-15T05:55:17.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 --2- 172.21.15.137:0/1179238506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b600f4be0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:17.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 -- 172.21.15.137:0/1179238506 >> 172.21.15.137:0/1179238506 conn(0x7f7b600f02e0 msgr2=0x7f7b600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:17.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.135+0000 7f7b65964700 1 -- 172.21.15.137:0/1179238506 shutdown_connections 2023-11-15T05:55:17.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.136+0000 7f7b65964700 1 -- 172.21.15.137:0/1179238506 wait complete. 2023-11-15T05:55:17.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.136+0000 7f7b65964700 1 Processor -- start 2023-11-15T05:55:17.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.136+0000 7f7b65964700 1 -- start start 2023-11-15T05:55:17.140 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.136+0000 7f7b65964700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:17.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.136+0000 7f7b65964700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b60108df0 con 0x7f7b600f47c0 2023-11-15T05:55:17.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b5effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:17.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b5effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55226/0 (socket says 172.21.15.137:55226) 2023-11-15T05:55:17.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b5effd700 1 -- 172.21.15.137:0/3205963341 learned_addr learned my addr 172.21.15.137:0/3205963341 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:17.141 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b5effd700 1 -- 172.21.15.137:0/3205963341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b54005040 con 0x7f7b600f47c0 2023-11-15T05:55:17.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b5effd700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7b54014000 tx=0x7f7b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:17.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.137+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b540095f0 con 0x7f7b600f47c0 2023-11-15T05:55:17.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b60108ff0 con 0x7f7b600f47c0 2023-11-15T05:55:17.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b54007e70 con 0x7f7b600f47c0 2023-11-15T05:55:17.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b54014ea0 con 0x7f7b600f47c0 2023-11-15T05:55:17.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b60109480 con 0x7f7b600f47c0 2023-11-15T05:55:17.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b54025070 con 0x7f7b600f47c0 2023-11-15T05:55:17.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.138+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b60045ad0 con 0x7f7b600f47c0 2023-11-15T05:55:17.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.139+0000 7f7b477fe700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:17.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.139+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b54040940 con 0x7f7b600f47c0 2023-11-15T05:55:17.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.139+0000 7f7b5e7fc700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:17.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.139+0000 7f7b5e7fc700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 0x7f7b48032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7b500096a0 tx=0x7f7b50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:17.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.144+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b540079c0 con 0x7f7b600f47c0 2023-11-15T05:55:17.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.350+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b6010a2d0 con 0x7f7b600f47c0 2023-11-15T05:55:17.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.351+0000 7f7b477fe700 1 -- 172.21.15.137:0/3205963341 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7b5400f300 con 0x7f7b600f47c0 2023-11-15T05:55:17.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:17.353 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:17.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 msgr2=0x7f7b48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:17.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 0x7f7b48032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f7b500096a0 tx=0x7f7b50006b40).stop 2023-11-15T05:55:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 msgr2=0x7f7b60108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7b54014000 tx=0x7f7b5400b040).stop 2023-11-15T05:55:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 shutdown_connections 2023-11-15T05:55:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b4802ff60 0x7f7b48032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 --2- 172.21.15.137:0/3205963341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b600f47c0 0x7f7b60108900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:17.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.352+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 >> 172.21.15.137:0/3205963341 conn(0x7f7b600f02e0 msgr2=0x7f7b600f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:17.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.353+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 shutdown_connections 2023-11-15T05:55:17.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:17.353+0000 7f7b65964700 1 -- 172.21.15.137:0/3205963341 wait complete. 2023-11-15T05:55:17.356 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:18.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:16.968259+0000 mgr.smithi003.cldzke (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:17.352358+0000 mon.smithi003 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.137:0/3205963341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:18.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:18.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:19.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- 172.21.15.137:0/2451198296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 msgr2=0x7f4dac0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:19.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 --2- 172.21.15.137:0/2451198296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac0fb550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4da0009fb0 tx=0x7f4da0009b30).stop 2023-11-15T05:55:19.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- 172.21.15.137:0/2451198296 shutdown_connections 2023-11-15T05:55:19.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 --2- 172.21.15.137:0/2451198296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac0fb550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:19.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- 172.21.15.137:0/2451198296 >> 172.21.15.137:0/2451198296 conn(0x7f4dac0f0b70 msgr2=0x7f4dac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:19.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- 172.21.15.137:0/2451198296 shutdown_connections 2023-11-15T05:55:19.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- 172.21.15.137:0/2451198296 wait complete. 2023-11-15T05:55:19.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 Processor -- start 2023-11-15T05:55:19.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.995+0000 7f4db13f8700 1 -- start start 2023-11-15T05:55:19.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4db13f8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:19.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4db13f8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac108da0 con 0x7f4dac060220 2023-11-15T05:55:19.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4daaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:19.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4daaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47332/0 (socket says 172.21.15.137:47332) 2023-11-15T05:55:19.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4daaffd700 1 -- 172.21.15.137:0/4083092021 learned_addr learned my addr 172.21.15.137:0/4083092021 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:20.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4daaffd700 1 -- 172.21.15.137:0/4083092021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0005040 con 0x7f4dac060220 2023-11-15T05:55:20.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.996+0000 7f4daaffd700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4da0013510 tx=0x7f4da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:20.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da0009320 con 0x7f4dac060220 2023-11-15T05:55:20.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac108fa0 con 0x7f4dac060220 2023-11-15T05:55:20.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4da0013f00 con 0x7f4dac060220 2023-11-15T05:55:20.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4da0014dc0 con 0x7f4dac060220 2023-11-15T05:55:20.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac109430 con 0x7f4dac060220 2023-11-15T05:55:20.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4da00095f0 con 0x7f4dac060220 2023-11-15T05:55:20.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.997+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac045ad0 con 0x7f4dac060220 2023-11-15T05:55:20.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.998+0000 7f4d937fe700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 0x7f4d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:20.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.998+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4da0041080 con 0x7f4dac060220 2023-11-15T05:55:20.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.998+0000 7f4daa7fc700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 0x7f4d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:20.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:19.998+0000 7f4daa7fc700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 0x7f4d94032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4d9c0096a0 tx=0x7f4d9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:20.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.002+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4da001c790 con 0x7f4dac060220 2023-11-15T05:55:20.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.207+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dac10a2e0 con 0x7f4dac060220 2023-11-15T05:55:20.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.208+0000 7f4d937fe700 1 -- 172.21.15.137:0/4083092021 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4da0010230 con 0x7f4dac060220 2023-11-15T05:55:20.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:20.209 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 msgr2=0x7f4d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 0x7f4d94032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4d9c0096a0 tx=0x7f4d9c006b40).stop 2023-11-15T05:55:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 msgr2=0x7f4dac1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4da0013510 tx=0x7f4da000b040).stop 2023-11-15T05:55:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 shutdown_connections 2023-11-15T05:55:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4d9402ff60 0x7f4d94032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 --2- 172.21.15.137:0/4083092021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4dac060220 0x7f4dac1088b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 >> 172.21.15.137:0/4083092021 conn(0x7f4dac0f0b70 msgr2=0x7f4dac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:20.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 shutdown_connections 2023-11-15T05:55:20.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:20.209+0000 7f4db13f8700 1 -- 172.21.15.137:0/4083092021 wait complete. 2023-11-15T05:55:20.212 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:20.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:18.968542+0000 mgr.smithi003.cldzke (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:21.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:20.208182+0000 mon.smithi003 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.137:0/4083092021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:21.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:21.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:22.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:20.968894+0000 mgr.smithi003.cldzke (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:22.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 -- 172.21.15.137:0/3664825126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 msgr2=0x7f9ee40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/3664825126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee40f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9ed8009fb0 tx=0x7f9ed8009b30).stop 2023-11-15T05:55:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 -- 172.21.15.137:0/3664825126 shutdown_connections 2023-11-15T05:55:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/3664825126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee40f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 -- 172.21.15.137:0/3664825126 >> 172.21.15.137:0/3664825126 conn(0x7f9ee40f0b70 msgr2=0x7f9ee40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 -- 172.21.15.137:0/3664825126 shutdown_connections 2023-11-15T05:55:22.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.789+0000 7f9eeaf55700 1 -- 172.21.15.137:0/3664825126 wait complete. 2023-11-15T05:55:22.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9eeaf55700 1 Processor -- start 2023-11-15T05:55:22.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9eeaf55700 1 -- start start 2023-11-15T05:55:22.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9eeaf55700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:22.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9eeaf55700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee4105b20 con 0x7f9ee40f5410 2023-11-15T05:55:22.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9ee8cf1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:22.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9ee8cf1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47352/0 (socket says 172.21.15.137:47352) 2023-11-15T05:55:22.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.790+0000 7f9ee8cf1700 1 -- 172.21.15.137:0/2260053409 learned_addr learned my addr 172.21.15.137:0/2260053409 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:22.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9ee8cf1700 1 -- 172.21.15.137:0/2260053409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8005040 con 0x7f9ee40f5410 2023-11-15T05:55:22.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9ee8cf1700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9ed8013510 tx=0x7f9ed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:22.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ed8009320 con 0x7f9ee40f5410 2023-11-15T05:55:22.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ed8013f00 con 0x7f9ee40f5410 2023-11-15T05:55:22.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ed8014d90 con 0x7f9ee40f5410 2023-11-15T05:55:22.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee4105d20 con 0x7f9ee40f5410 2023-11-15T05:55:22.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.791+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee41061b0 con 0x7f9ee40f5410 2023-11-15T05:55:22.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.792+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ed8014420 con 0x7f9ee40f5410 2023-11-15T05:55:22.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.792+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee4045ad0 con 0x7f9ee40f5410 2023-11-15T05:55:22.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.792+0000 7f9ee17fa700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 0x7f9ed4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:22.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.792+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ed8041090 con 0x7f9ee40f5410 2023-11-15T05:55:22.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.792+0000 7f9ee3fff700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 0x7f9ed4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:22.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.793+0000 7f9ee3fff700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 0x7f9ed4032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9ed00096a0 tx=0x7f9ed0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:22.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:22.797+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ed806d2b0 con 0x7f9ee40f5410 2023-11-15T05:55:23.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.002+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ee40fae60 con 0x7f9ee40f5410 2023-11-15T05:55:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.002+0000 7f9ee17fa700 1 -- 172.21.15.137:0/2260053409 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9ed80095f0 con 0x7f9ee40f5410 2023-11-15T05:55:23.003 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:23.003 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 msgr2=0x7f9ed4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 0x7f9ed4032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9ed00096a0 tx=0x7f9ed0006b40).stop 2023-11-15T05:55:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 msgr2=0x7f9ee4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9ed8013510 tx=0x7f9ed800b040).stop 2023-11-15T05:55:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 shutdown_connections 2023-11-15T05:55:23.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ed402ffb0 0x7f9ed4032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 --2- 172.21.15.137:0/2260053409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee40f5410 0x7f9ee4105630 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 >> 172.21.15.137:0/2260053409 conn(0x7f9ee40f0b70 msgr2=0x7f9ee40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 shutdown_connections 2023-11-15T05:55:23.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:23.004+0000 7f9eeaf55700 1 -- 172.21.15.137:0/2260053409 wait complete. 2023-11-15T05:55:23.007 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:23.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:23.002602+0000 mon.smithi003 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.137:0/2260053409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:24.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:24.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:24.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:22.969273+0000 mgr.smithi003.cldzke (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:24.969661+0000 mgr.smithi003.cldzke (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:26.970064+0000 mgr.smithi003.cldzke (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:28.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 -- 172.21.15.137:0/2075882639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 msgr2=0x7fa1d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/2075882639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d80f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fa1c8009fb0 tx=0x7fa1c8009b30).stop 2023-11-15T05:55:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 -- 172.21.15.137:0/2075882639 shutdown_connections 2023-11-15T05:55:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/2075882639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d80f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 -- 172.21.15.137:0/2075882639 >> 172.21.15.137:0/2075882639 conn(0x7fa1d80f0b70 msgr2=0x7fa1d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:28.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 -- 172.21.15.137:0/2075882639 shutdown_connections 2023-11-15T05:55:28.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.484+0000 7fa1df4b8700 1 -- 172.21.15.137:0/2075882639 wait complete. 2023-11-15T05:55:28.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1df4b8700 1 Processor -- start 2023-11-15T05:55:28.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1df4b8700 1 -- start start 2023-11-15T05:55:28.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1df4b8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:28.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1df4b8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d8060220 con 0x7fa1d80f5410 2023-11-15T05:55:28.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1dd254700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:28.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1dd254700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47378/0 (socket says 172.21.15.137:47378) 2023-11-15T05:55:28.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1dd254700 1 -- 172.21.15.137:0/756774111 learned_addr learned my addr 172.21.15.137:0/756774111 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:28.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.485+0000 7fa1dd254700 1 -- 172.21.15.137:0/756774111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8005040 con 0x7fa1d80f5410 2023-11-15T05:55:28.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1dd254700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa1c8014000 tx=0x7fa1c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:28.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1c80095f0 con 0x7fa1d80f5410 2023-11-15T05:55:28.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d8060420 con 0x7fa1d80f5410 2023-11-15T05:55:28.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1c8007e70 con 0x7fa1d80f5410 2023-11-15T05:55:28.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1c8014ea0 con 0x7fa1d80f5410 2023-11-15T05:55:28.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.486+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d8109b00 con 0x7fa1d80f5410 2023-11-15T05:55:28.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.487+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1c8025070 con 0x7fa1d80f5410 2023-11-15T05:55:28.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.487+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d8059100 con 0x7fa1d80f5410 2023-11-15T05:55:28.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.487+0000 7fa1cdffb700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 0x7fa1c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.487+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1c8040930 con 0x7fa1d80f5410 2023-11-15T05:55:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.487+0000 7fa1dca53700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 0x7fa1c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.488+0000 7fa1dca53700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 0x7fa1c4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fa1d40096a0 tx=0x7fa1d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.492+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1c800f4e0 con 0x7fa1d80f5410 2023-11-15T05:55:28.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.702+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1d8045ad0 con 0x7fa1d80f5410 2023-11-15T05:55:28.703 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.703+0000 7fa1cdffb700 1 -- 172.21.15.137:0/756774111 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa1c800f300 con 0x7fa1d80f5410 2023-11-15T05:55:28.704 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:28.704 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:28.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 msgr2=0x7fa1c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:28.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 0x7fa1c4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fa1d40096a0 tx=0x7fa1d4006b40).stop 2023-11-15T05:55:28.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 msgr2=0x7fa1d8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:28.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa1c8014000 tx=0x7fa1c800b040).stop 2023-11-15T05:55:28.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 shutdown_connections 2023-11-15T05:55:28.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa1c402ff60 0x7fa1c4032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:28.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 --2- 172.21.15.137:0/756774111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1d80f5410 0x7fa1d8109450 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:28.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 >> 172.21.15.137:0/756774111 conn(0x7fa1d80f0b70 msgr2=0x7fa1d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:28.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 shutdown_connections 2023-11-15T05:55:28.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:28.705+0000 7fa1df4b8700 1 -- 172.21.15.137:0/756774111 wait complete. 2023-11-15T05:55:28.708 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:29.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:29 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:28.703163+0000 mon.smithi003 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.137:0/756774111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:30.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:28.970412+0000 mgr.smithi003.cldzke (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:31.266 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:31.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:31.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:30.970800+0000 mgr.smithi003.cldzke (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:34.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:32.971122+0000 mgr.smithi003.cldzke (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:35.173 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 -- 172.21.15.137:0/3517486939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 msgr2=0x7fb9440f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:35.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 --2- 172.21.15.137:0/3517486939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9440f5750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb938009fb0 tx=0x7fb938009b30).stop 2023-11-15T05:55:35.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 -- 172.21.15.137:0/3517486939 shutdown_connections 2023-11-15T05:55:35.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 --2- 172.21.15.137:0/3517486939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9440f5750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 -- 172.21.15.137:0/3517486939 >> 172.21.15.137:0/3517486939 conn(0x7fb9440f0a90 msgr2=0x7fb9440f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 -- 172.21.15.137:0/3517486939 shutdown_connections 2023-11-15T05:55:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.172+0000 7fb94b154700 1 -- 172.21.15.137:0/3517486939 wait complete. 2023-11-15T05:55:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.173+0000 7fb94b154700 1 Processor -- start 2023-11-15T05:55:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.173+0000 7fb94b154700 1 -- start start 2023-11-15T05:55:35.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.173+0000 7fb94b154700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:35.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.173+0000 7fb94b154700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944105dc0 con 0x7fb9440f5330 2023-11-15T05:55:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb948ef0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb948ef0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48302/0 (socket says 172.21.15.137:48302) 2023-11-15T05:55:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb948ef0700 1 -- 172.21.15.137:0/3277308471 learned_addr learned my addr 172.21.15.137:0/3277308471 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb948ef0700 1 -- 172.21.15.137:0/3277308471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb938005040 con 0x7fb9440f5330 2023-11-15T05:55:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb948ef0700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb938013510 tx=0x7fb93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb938009320 con 0x7fb9440f5330 2023-11-15T05:55:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb938013f00 con 0x7fb9440f5330 2023-11-15T05:55:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944105fc0 con 0x7fb9440f5330 2023-11-15T05:55:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb938014d90 con 0x7fb9440f5330 2023-11-15T05:55:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.174+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944106450 con 0x7fb9440f5330 2023-11-15T05:55:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.175+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9380095f0 con 0x7fb9440f5330 2023-11-15T05:55:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.175+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb944045ad0 con 0x7fb9440f5330 2023-11-15T05:55:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.175+0000 7fb9417fa700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 0x7fb934032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.175+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb938042dc0 con 0x7fb9440f5330 2023-11-15T05:55:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.176+0000 7fb943fff700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 0x7fb934032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:35.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.176+0000 7fb943fff700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 0x7fb934032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb9300096a0 tx=0x7fb930006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:35.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.180+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb938014420 con 0x7fb9440f5330 2023-11-15T05:55:35.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.386+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb94405ea50 con 0x7fb9440f5330 2023-11-15T05:55:35.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.387+0000 7fb9417fa700 1 -- 172.21.15.137:0/3277308471 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb938011290 con 0x7fb9440f5330 2023-11-15T05:55:35.389 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:35.389 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:35.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 msgr2=0x7fb934032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:35.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 0x7fb934032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb9300096a0 tx=0x7fb930006b40).stop 2023-11-15T05:55:35.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 msgr2=0x7fb9441058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:35.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb938013510 tx=0x7fb93800b040).stop 2023-11-15T05:55:35.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 shutdown_connections 2023-11-15T05:55:35.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb93402ffb0 0x7fb934032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:35.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 --2- 172.21.15.137:0/3277308471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9440f5330 0x7fb9441058d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:35.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 >> 172.21.15.137:0/3277308471 conn(0x7fb9440f0a90 msgr2=0x7fb9440f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:35.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 shutdown_connections 2023-11-15T05:55:35.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:35.389+0000 7fb94b154700 1 -- 172.21.15.137:0/3277308471 wait complete. 2023-11-15T05:55:35.392 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:34.971454+0000 mgr.smithi003.cldzke (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:35.109997+0000 mon.smithi003 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:55:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:35.113562+0000 mon.smithi003 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:55:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:35.387554+0000 mon.smithi003 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.137:0/3277308471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:36.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:36.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:37.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- 172.21.15.137:0/1247569041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 msgr2=0x7efff40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:37.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 --2- 172.21.15.137:0/1247569041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff40f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7effe4009fb0 tx=0x7effe4009b30).stop 2023-11-15T05:55:37.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- 172.21.15.137:0/1247569041 shutdown_connections 2023-11-15T05:55:37.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 --2- 172.21.15.137:0/1247569041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff40f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:37.926 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- 172.21.15.137:0/1247569041 >> 172.21.15.137:0/1247569041 conn(0x7efff40f0b70 msgr2=0x7efff40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:37.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- 172.21.15.137:0/1247569041 shutdown_connections 2023-11-15T05:55:37.927 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- 172.21.15.137:0/1247569041 wait complete. 2023-11-15T05:55:37.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 Processor -- start 2023-11-15T05:55:37.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.925+0000 7efffb585700 1 -- start start 2023-11-15T05:55:37.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efffb585700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:37.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efffb585700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff4105b20 con 0x7efff40f5410 2023-11-15T05:55:37.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efff9321700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:37.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efff9321700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48326/0 (socket says 172.21.15.137:48326) 2023-11-15T05:55:37.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efff9321700 1 -- 172.21.15.137:0/1876245895 learned_addr learned my addr 172.21.15.137:0/1876245895 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efff9321700 1 -- 172.21.15.137:0/1876245895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4005040 con 0x7efff40f5410 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efff9321700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7effe4013510 tx=0x7effe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe4009320 con 0x7efff40f5410 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effe4013f00 con 0x7efff40f5410 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effe4014d90 con 0x7efff40f5410 2023-11-15T05:55:37.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.926+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff4105d20 con 0x7efff40f5410 2023-11-15T05:55:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.927+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff41061b0 con 0x7efff40f5410 2023-11-15T05:55:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.927+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff4045ad0 con 0x7efff40f5410 2023-11-15T05:55:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.927+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effe40095f0 con 0x7efff40f5410 2023-11-15T05:55:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.928+0000 7effe9ffb700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.928+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effe4040f90 con 0x7efff40f5410 2023-11-15T05:55:37.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.928+0000 7efff8b20700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:37.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.928+0000 7efff8b20700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 0x7effe0032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:37.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:37.932+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effe4014420 con 0x7efff40f5410 2023-11-15T05:55:38.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.138+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efff40fae60 con 0x7efff40f5410 2023-11-15T05:55:38.139 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.139+0000 7effe9ffb700 1 -- 172.21.15.137:0/1876245895 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7effe401d7f0 con 0x7efff40f5410 2023-11-15T05:55:38.140 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:38.140 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:38.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 msgr2=0x7effe0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:38.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 0x7effe0032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7efff00096a0 tx=0x7efff0006b40).stop 2023-11-15T05:55:38.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 msgr2=0x7efff4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:38.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7effe4013510 tx=0x7effe400b040).stop 2023-11-15T05:55:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 shutdown_connections 2023-11-15T05:55:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7effe002ffb0 0x7effe0032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 --2- 172.21.15.137:0/1876245895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff40f5410 0x7efff4105630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 >> 172.21.15.137:0/1876245895 conn(0x7efff40f0b70 msgr2=0x7efff40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:38.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 shutdown_connections 2023-11-15T05:55:38.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:38.140+0000 7efffb585700 1 -- 172.21.15.137:0/1876245895 wait complete. 2023-11-15T05:55:38.144 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:36.971866+0000 mgr.smithi003.cldzke (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:39.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:38.139239+0000 mon.smithi003 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.137:0/1876245895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:39.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:39.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:38.972132+0000 mgr.smithi003.cldzke (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:40.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 -- 172.21.15.137:0/1455533327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 msgr2=0x7f5eec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:40.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 --2- 172.21.15.137:0/1455533327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f5edc009fb0 tx=0x7f5edc009b30).stop 2023-11-15T05:55:40.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 -- 172.21.15.137:0/1455533327 shutdown_connections 2023-11-15T05:55:40.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 --2- 172.21.15.137:0/1455533327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:40.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 -- 172.21.15.137:0/1455533327 >> 172.21.15.137:0/1455533327 conn(0x7f5eec0f0b70 msgr2=0x7f5eec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 -- 172.21.15.137:0/1455533327 shutdown_connections 2023-11-15T05:55:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.660+0000 7f5ef2551700 1 -- 172.21.15.137:0/1455533327 wait complete. 2023-11-15T05:55:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5ef2551700 1 Processor -- start 2023-11-15T05:55:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5ef2551700 1 -- start start 2023-11-15T05:55:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5ef2551700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:40.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5ef2551700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec060220 con 0x7f5eec0f5410 2023-11-15T05:55:40.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5eebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:40.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5eebfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39036/0 (socket says 172.21.15.137:39036) 2023-11-15T05:55:40.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5eebfff700 1 -- 172.21.15.137:0/529628936 learned_addr learned my addr 172.21.15.137:0/529628936 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:40.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.661+0000 7f5eebfff700 1 -- 172.21.15.137:0/529628936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc005040 con 0x7f5eec0f5410 2023-11-15T05:55:40.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5eebfff700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5edc013510 tx=0x7f5edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:40.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc009320 con 0x7f5eec0f5410 2023-11-15T05:55:40.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec060420 con 0x7f5eec0f5410 2023-11-15T05:55:40.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5edc013f00 con 0x7f5eec0f5410 2023-11-15T05:55:40.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc014d90 con 0x7f5eec0f5410 2023-11-15T05:55:40.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.662+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec109aa0 con 0x7f5eec0f5410 2023-11-15T05:55:40.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.663+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5edc014420 con 0x7f5eec0f5410 2023-11-15T05:55:40.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.663+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec059100 con 0x7f5eec0f5410 2023-11-15T05:55:40.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.663+0000 7f5ee8ff9700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:40.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.663+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5edc040fb0 con 0x7f5eec0f5410 2023-11-15T05:55:40.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.663+0000 7f5eeb7fe700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:40.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.664+0000 7f5eeb7fe700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 0x7f5ed4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5ee00096a0 tx=0x7f5ee0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:40.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.668+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5edc0095f0 con 0x7f5eec0f5410 2023-11-15T05:55:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.874+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5eec045ad0 con 0x7f5eec0f5410 2023-11-15T05:55:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.875+0000 7f5ee8ff9700 1 -- 172.21.15.137:0/529628936 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5edc01db70 con 0x7f5eec0f5410 2023-11-15T05:55:40.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:40.877 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:40.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 msgr2=0x7f5ed4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:40.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 0x7f5ed4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5ee00096a0 tx=0x7f5ee0006b40).stop 2023-11-15T05:55:40.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 msgr2=0x7f5eec1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:40.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f5edc013510 tx=0x7f5edc00b040).stop 2023-11-15T05:55:40.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 shutdown_connections 2023-11-15T05:55:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 --2- 172.21.15.137:0/529628936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5eec0f5410 0x7f5eec1093f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 >> 172.21.15.137:0/529628936 conn(0x7f5eec0f0b70 msgr2=0x7f5eec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 shutdown_connections 2023-11-15T05:55:40.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:40.877+0000 7f5ef2551700 1 -- 172.21.15.137:0/529628936 wait complete. 2023-11-15T05:55:40.881 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:40.876378+0000 mon.smithi003 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.137:0/529628936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:40.972495+0000 mgr.smithi003.cldzke (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:42.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:42.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:43.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 -- 172.21.15.137:0/1574120980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 msgr2=0x7f72900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:43.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 --2- 172.21.15.137:0/1574120980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f72900f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f7284009fb0 tx=0x7f7284009b30).stop 2023-11-15T05:55:43.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 -- 172.21.15.137:0/1574120980 shutdown_connections 2023-11-15T05:55:43.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 --2- 172.21.15.137:0/1574120980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f72900f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:43.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 -- 172.21.15.137:0/1574120980 >> 172.21.15.137:0/1574120980 conn(0x7f72900f0b70 msgr2=0x7f72900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:43.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 -- 172.21.15.137:0/1574120980 shutdown_connections 2023-11-15T05:55:43.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.432+0000 7f72958bd700 1 -- 172.21.15.137:0/1574120980 wait complete. 2023-11-15T05:55:43.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f72958bd700 1 Processor -- start 2023-11-15T05:55:43.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f72958bd700 1 -- start start 2023-11-15T05:55:43.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f72958bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:43.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f72958bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290107240 con 0x7f72900f5410 2023-11-15T05:55:43.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f728effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:43.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f728effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39048/0 (socket says 172.21.15.137:39048) 2023-11-15T05:55:43.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f728effd700 1 -- 172.21.15.137:0/3587442424 learned_addr learned my addr 172.21.15.137:0/3587442424 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:43.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.433+0000 7f728effd700 1 -- 172.21.15.137:0/3587442424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284005040 con 0x7f72900f5410 2023-11-15T05:55:43.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f728effd700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7284013510 tx=0x7f728400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:43.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7284009320 con 0x7f72900f5410 2023-11-15T05:55:43.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7290060220 con 0x7f72900f5410 2023-11-15T05:55:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7284013f00 con 0x7f72900f5410 2023-11-15T05:55:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729010bd40 con 0x7f72900f5410 2023-11-15T05:55:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.434+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7284014d90 con 0x7f72900f5410 2023-11-15T05:55:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72840095f0 con 0x7f72900f5410 2023-11-15T05:55:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290045ad0 con 0x7f72900f5410 2023-11-15T05:55:43.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f72777fe700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 0x7f7278032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:43.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7284041090 con 0x7f72900f5410 2023-11-15T05:55:43.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f728e7fc700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 0x7f7278032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:43.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.435+0000 7f728e7fc700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 0x7f7278032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f72800096a0 tx=0x7f7280006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:43.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.439+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7284014420 con 0x7f72900f5410 2023-11-15T05:55:43.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.647+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72900fae60 con 0x7f72900f5410 2023-11-15T05:55:43.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.647+0000 7f72777fe700 1 -- 172.21.15.137:0/3587442424 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7284010230 con 0x7f72900f5410 2023-11-15T05:55:43.649 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:43.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 msgr2=0x7f7278032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:43.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 0x7f7278032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f72800096a0 tx=0x7f7280006b40).stop 2023-11-15T05:55:43.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 msgr2=0x7f729010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:43.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f7284013510 tx=0x7f728400b040).stop 2023-11-15T05:55:43.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 shutdown_connections 2023-11-15T05:55:43.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f727802ff60 0x7f7278032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:43.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 --2- 172.21.15.137:0/3587442424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72900f5410 0x7f729010b850 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:43.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 >> 172.21.15.137:0/3587442424 conn(0x7f72900f0b70 msgr2=0x7f72900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:43.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 shutdown_connections 2023-11-15T05:55:43.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:43.649+0000 7f72958bd700 1 -- 172.21.15.137:0/3587442424 wait complete. 2023-11-15T05:55:43.653 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:44.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:42.972755+0000 mgr.smithi003.cldzke (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:43.648924+0000 mon.smithi003 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.137:0/3587442424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:45.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:45.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:46.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 -- 172.21.15.137:0/1169685370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 msgr2=0x7fbc040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:46.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 --2- 172.21.15.137:0/1169685370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc040f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbbf8009fb0 tx=0x7fbbf8009b30).stop 2023-11-15T05:55:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 -- 172.21.15.137:0/1169685370 shutdown_connections 2023-11-15T05:55:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 --2- 172.21.15.137:0/1169685370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc040f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 -- 172.21.15.137:0/1169685370 >> 172.21.15.137:0/1169685370 conn(0x7fbc040f1c70 msgr2=0x7fbc040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 -- 172.21.15.137:0/1169685370 shutdown_connections 2023-11-15T05:55:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 -- 172.21.15.137:0/1169685370 wait complete. 2023-11-15T05:55:46.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.219+0000 7fbc0a363700 1 Processor -- start 2023-11-15T05:55:46.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc0a363700 1 -- start start 2023-11-15T05:55:46.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc0a363700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:46.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc0a363700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04069e20 con 0x7fbc040f6510 2023-11-15T05:55:46.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39066/0 (socket says 172.21.15.137:39066) 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc03fff700 1 -- 172.21.15.137:0/616740430 learned_addr learned my addr 172.21.15.137:0/616740430 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc03fff700 1 -- 172.21.15.137:0/616740430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8005040 con 0x7fbc040f6510 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc03fff700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbbf8013510 tx=0x7fbbf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.220+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8009320 con 0x7fbc040f6510 2023-11-15T05:55:46.225 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc04069fc0 con 0x7fbc040f6510 2023-11-15T05:55:46.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbf8013f00 con 0x7fbc040f6510 2023-11-15T05:55:46.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8014d90 con 0x7fbc040f6510 2023-11-15T05:55:46.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0406a450 con 0x7fbc040f6510 2023-11-15T05:55:46.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbf80095f0 con 0x7fbc040f6510 2023-11-15T05:55:46.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.221+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc04059100 con 0x7fbc040f6510 2023-11-15T05:55:46.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.222+0000 7fbc00ff9700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:46.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.222+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbf8040fc0 con 0x7fbc040f6510 2023-11-15T05:55:46.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.222+0000 7fbc037fe700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:46.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.222+0000 7fbc037fe700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 0x7fbbec032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fbbf40096a0 tx=0x7fbbf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:46.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.226+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbf8014420 con 0x7fbc040f6510 2023-11-15T05:55:46.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:44.973020+0000 mgr.smithi003.cldzke (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:46.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.432+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc04045ad0 con 0x7fbc040f6510 2023-11-15T05:55:46.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.433+0000 7fbc00ff9700 1 -- 172.21.15.137:0/616740430 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbbf8010230 con 0x7fbc040f6510 2023-11-15T05:55:46.435 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:46.435 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:46.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 msgr2=0x7fbbec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:46.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 0x7fbbec032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fbbf40096a0 tx=0x7fbbf4006b40).stop 2023-11-15T05:55:46.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 msgr2=0x7fbc0406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:46.437 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fbbf8013510 tx=0x7fbbf800b040).stop 2023-11-15T05:55:46.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 shutdown_connections 2023-11-15T05:55:46.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:46.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 --2- 172.21.15.137:0/616740430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbc040f6510 0x7fbc0406b9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:46.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 >> 172.21.15.137:0/616740430 conn(0x7fbc040f1c70 msgr2=0x7fbc040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:46.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 shutdown_connections 2023-11-15T05:55:46.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:46.435+0000 7fbc0a363700 1 -- 172.21.15.137:0/616740430 wait complete. 2023-11-15T05:55:46.439 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:47.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:46.434392+0000 mon.smithi003 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.137:0/616740430' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:47.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:47.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:48.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:46.973280+0000 mgr.smithi003.cldzke (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:49.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- 172.21.15.137:0/1291046744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 msgr2=0x7f90400fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:49.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 --2- 172.21.15.137:0/1291046744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90400fb570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9030009fb0 tx=0x7f9030009b30).stop 2023-11-15T05:55:49.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- 172.21.15.137:0/1291046744 shutdown_connections 2023-11-15T05:55:49.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 --2- 172.21.15.137:0/1291046744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90400fb570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:49.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- 172.21.15.137:0/1291046744 >> 172.21.15.137:0/1291046744 conn(0x7f90400f0b70 msgr2=0x7f90400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:49.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- 172.21.15.137:0/1291046744 shutdown_connections 2023-11-15T05:55:49.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- 172.21.15.137:0/1291046744 wait complete. 2023-11-15T05:55:49.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 Processor -- start 2023-11-15T05:55:49.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- start start 2023-11-15T05:55:49.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:49.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.006+0000 7f9047886700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9040108dc0 con 0x7f9040060200 2023-11-15T05:55:49.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9045622700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9045622700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39086/0 (socket says 172.21.15.137:39086) 2023-11-15T05:55:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9045622700 1 -- 172.21.15.137:0/838579644 learned_addr learned my addr 172.21.15.137:0/838579644 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9045622700 1 -- 172.21.15.137:0/838579644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030005040 con 0x7f9040060200 2023-11-15T05:55:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9045622700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9030013510 tx=0x7f903000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9030009320 con 0x7f9040060200 2023-11-15T05:55:49.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9040108fc0 con 0x7f9040060200 2023-11-15T05:55:49.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9030013f00 con 0x7f9040060200 2023-11-15T05:55:49.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9040109450 con 0x7f9040060200 2023-11-15T05:55:49.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.007+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9030014de0 con 0x7f9040060200 2023-11-15T05:55:49.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.008+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f903001e420 con 0x7f9040060200 2023-11-15T05:55:49.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.008+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9040045ad0 con 0x7f9040060200 2023-11-15T05:55:49.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.008+0000 7f9035ffb700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 0x7f902c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:49.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.009+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90300412b0 con 0x7f9040060200 2023-11-15T05:55:49.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.009+0000 7f9044e21700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 0x7f902c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:49.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.009+0000 7f9044e21700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 0x7f902c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f903c0096a0 tx=0x7f903c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:49.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.014+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f903001c320 con 0x7f9040060200 2023-11-15T05:55:49.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.218+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f904010a300 con 0x7f9040060200 2023-11-15T05:55:49.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.218+0000 7f9035ffb700 1 -- 172.21.15.137:0/838579644 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f903002b2c0 con 0x7f9040060200 2023-11-15T05:55:49.221 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:49.221 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:49.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 msgr2=0x7f902c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:49.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 0x7f902c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f903c0096a0 tx=0x7f903c006b40).stop 2023-11-15T05:55:49.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 msgr2=0x7f90401088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:49.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9030013510 tx=0x7f903000b040).stop 2023-11-15T05:55:49.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 shutdown_connections 2023-11-15T05:55:49.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f902c02ff60 0x7f902c032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:49.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 --2- 172.21.15.137:0/838579644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9040060200 0x7f90401088d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:49.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 >> 172.21.15.137:0/838579644 conn(0x7f90400f0b70 msgr2=0x7f90400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:49.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 shutdown_connections 2023-11-15T05:55:49.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:49.220+0000 7f9047886700 1 -- 172.21.15.137:0/838579644 wait complete. 2023-11-15T05:55:49.224 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:48.973621+0000 mgr.smithi003.cldzke (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:49.220008+0000 mon.smithi003 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.137:0/838579644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:50.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:50.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:51.809 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 -- 172.21.15.137:0/790374550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 msgr2=0x7f744c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 --2- 172.21.15.137:0/790374550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c0f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f743c009fb0 tx=0x7f743c009b30).stop 2023-11-15T05:55:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 -- 172.21.15.137:0/790374550 shutdown_connections 2023-11-15T05:55:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 --2- 172.21.15.137:0/790374550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c0f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 -- 172.21.15.137:0/790374550 >> 172.21.15.137:0/790374550 conn(0x7f744c0f1c70 msgr2=0x7f744c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 -- 172.21.15.137:0/790374550 shutdown_connections 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.807+0000 7f7452713700 1 -- 172.21.15.137:0/790374550 wait complete. 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f7452713700 1 Processor -- start 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f7452713700 1 -- start start 2023-11-15T05:55:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f7452713700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:51.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f7452713700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c069e20 con 0x7f744c0f6510 2023-11-15T05:55:51.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f744bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f744bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:41368/0 (socket says 172.21.15.137:41368) 2023-11-15T05:55:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.808+0000 7f744bfff700 1 -- 172.21.15.137:0/2568059424 learned_addr learned my addr 172.21.15.137:0/2568059424 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f744bfff700 1 -- 172.21.15.137:0/2568059424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c005040 con 0x7f744c0f6510 2023-11-15T05:55:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f744bfff700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f743c013510 tx=0x7f743c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c009320 con 0x7f744c0f6510 2023-11-15T05:55:51.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c069fc0 con 0x7f744c0f6510 2023-11-15T05:55:51.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744c06a450 con 0x7f744c0f6510 2023-11-15T05:55:51.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f743c013f00 con 0x7f744c0f6510 2023-11-15T05:55:51.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.809+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c014d60 con 0x7f744c0f6510 2023-11-15T05:55:51.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.810+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f743c014420 con 0x7f744c0f6510 2023-11-15T05:55:51.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.810+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744c059100 con 0x7f744c0f6510 2023-11-15T05:55:51.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.810+0000 7f7448ff9700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:51.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.810+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f743c041360 con 0x7f744c0f6510 2023-11-15T05:55:51.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.810+0000 7f744b7fe700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:51.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.811+0000 7f744b7fe700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 0x7f7434032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f74400096a0 tx=0x7f7440006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:51.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:51.815+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f743c01c320 con 0x7f744c0f6510 2023-11-15T05:55:51.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:50.973991+0000 mgr.smithi003.cldzke (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:52.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.022+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f744c0fbf60 con 0x7f744c0f6510 2023-11-15T05:55:52.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.023+0000 7f7448ff9700 1 -- 172.21.15.137:0/2568059424 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f743c012b20 con 0x7f744c0f6510 2023-11-15T05:55:52.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:52.025 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:52.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 msgr2=0x7f7434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:52.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 0x7f7434032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f74400096a0 tx=0x7f7440006b40).stop 2023-11-15T05:55:52.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 msgr2=0x7f744c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:52.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f743c013510 tx=0x7f743c00b040).stop 2023-11-15T05:55:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 shutdown_connections 2023-11-15T05:55:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f743402ff60 0x7f7434032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 --2- 172.21.15.137:0/2568059424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f744c0f6510 0x7f744c06b9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 >> 172.21.15.137:0/2568059424 conn(0x7f744c0f1c70 msgr2=0x7f744c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 shutdown_connections 2023-11-15T05:55:52.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:52.024+0000 7f7452713700 1 -- 172.21.15.137:0/2568059424 wait complete. 2023-11-15T05:55:52.028 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:52.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:52.024164+0000 mon.smithi003 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.137:0/2568059424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:53.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:53.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:53.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:52.974327+0000 mgr.smithi003.cldzke (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:54.637 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 -- 172.21.15.137:0/3225751152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 msgr2=0x7ff8b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:54.637 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 --2- 172.21.15.137:0/3225751152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b00f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff8a0009fb0 tx=0x7ff8a0009b30).stop 2023-11-15T05:55:54.637 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 -- 172.21.15.137:0/3225751152 shutdown_connections 2023-11-15T05:55:54.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 --2- 172.21.15.137:0/3225751152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b00f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:54.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 -- 172.21.15.137:0/3225751152 >> 172.21.15.137:0/3225751152 conn(0x7ff8b00f1aa0 msgr2=0x7ff8b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:54.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 -- 172.21.15.137:0/3225751152 shutdown_connections 2023-11-15T05:55:54.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.634+0000 7ff8af59e700 1 -- 172.21.15.137:0/3225751152 wait complete. 2023-11-15T05:55:54.638 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8af59e700 1 Processor -- start 2023-11-15T05:55:54.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8af59e700 1 -- start start 2023-11-15T05:55:54.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8af59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:54.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8af59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b01069a0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:54.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8ae59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:41392/0 (socket says 172.21.15.137:41392) 2023-11-15T05:55:54.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8ae59c700 1 -- 172.21.15.137:0/2708285347 learned_addr learned my addr 172.21.15.137:0/2708285347 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:54.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.635+0000 7ff8ae59c700 1 -- 172.21.15.137:0/2708285347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0005040 con 0x7ff8b00f62a0 2023-11-15T05:55:54.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff8ae59c700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff8a0009830 tx=0x7ff8a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:54.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a0009320 con 0x7ff8b00f62a0 2023-11-15T05:55:54.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8a00136c0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b0106ba0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a001ee90 con 0x7ff8b00f62a0 2023-11-15T05:55:54.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.636+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b0107030 con 0x7ff8b00f62a0 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.637+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8a001e420 con 0x7ff8b00f62a0 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.637+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b0045ad0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.637+0000 7ff89effd700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 0x7ff8980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.637+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8a00410b0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.637+0000 7ff8add9b700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 0x7ff8980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.638+0000 7ff8add9b700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 0x7ff8980323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff8a40096a0 tx=0x7ff8a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:54.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.642+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8a00095f0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.847 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.845+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8b0107e60 con 0x7ff8b00f62a0 2023-11-15T05:55:54.848 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.846+0000 7ff89effd700 1 -- 172.21.15.137:0/2708285347 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff8a002c2c0 con 0x7ff8b00f62a0 2023-11-15T05:55:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:54.848 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:54.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 msgr2=0x7ff8980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:54.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 0x7ff8980323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff8a40096a0 tx=0x7ff8a4006b40).stop 2023-11-15T05:55:54.850 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 msgr2=0x7ff8b01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:54.850 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff8a0009830 tx=0x7ff8a000b040).stop 2023-11-15T05:55:54.850 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 shutdown_connections 2023-11-15T05:55:54.850 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff89802ff10 0x7ff8980323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:54.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 --2- 172.21.15.137:0/2708285347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b00f62a0 0x7ff8b01064b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:54.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 >> 172.21.15.137:0/2708285347 conn(0x7ff8b00f1aa0 msgr2=0x7ff8b00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:54.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 shutdown_connections 2023-11-15T05:55:54.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:54.848+0000 7ff8af59e700 1 -- 172.21.15.137:0/2708285347 wait complete. 2023-11-15T05:55:54.852 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:55.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:54.847627+0000 mon.smithi003 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.137:0/2708285347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:56.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:55 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:54.974579+0000 mgr.smithi003.cldzke (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:56.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:56.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:55:57.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 -- 172.21.15.137:0/3475830367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 msgr2=0x7f478c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:57.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 --2- 172.21.15.137:0/3475830367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c0f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f477c009fb0 tx=0x7f477c009b30).stop 2023-11-15T05:55:57.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 -- 172.21.15.137:0/3475830367 shutdown_connections 2023-11-15T05:55:57.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 --2- 172.21.15.137:0/3475830367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c0f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:57.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 -- 172.21.15.137:0/3475830367 >> 172.21.15.137:0/3475830367 conn(0x7f478c0f0b70 msgr2=0x7f478c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:57.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 -- 172.21.15.137:0/3475830367 shutdown_connections 2023-11-15T05:55:57.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 -- 172.21.15.137:0/3475830367 wait complete. 2023-11-15T05:55:57.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.438+0000 7f479254e700 1 Processor -- start 2023-11-15T05:55:57.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f479254e700 1 -- start start 2023-11-15T05:55:57.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f479254e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f479254e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c060220 con 0x7f478c0f5410 2023-11-15T05:55:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f478bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f478bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:41400/0 (socket says 172.21.15.137:41400) 2023-11-15T05:55:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f478bfff700 1 -- 172.21.15.137:0/39160987 learned_addr learned my addr 172.21.15.137:0/39160987 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:55:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f478bfff700 1 -- 172.21.15.137:0/39160987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c005040 con 0x7f478c0f5410 2023-11-15T05:55:57.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.439+0000 7f478bfff700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f477c013510 tx=0x7f477c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:57.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.440+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c009320 con 0x7f478c0f5410 2023-11-15T05:55:57.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.440+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c060420 con 0x7f478c0f5410 2023-11-15T05:55:57.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.440+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f477c013f00 con 0x7f478c0f5410 2023-11-15T05:55:57.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.440+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f477c014d90 con 0x7f478c0f5410 2023-11-15T05:55:57.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.440+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c109b00 con 0x7f478c0f5410 2023-11-15T05:55:57.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.441+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c059100 con 0x7f478c0f5410 2023-11-15T05:55:57.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.441+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f477c0095f0 con 0x7f478c0f5410 2023-11-15T05:55:57.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.441+0000 7f4788ff9700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 0x7f4778032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:55:57.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.441+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f477c049920 con 0x7f478c0f5410 2023-11-15T05:55:57.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.441+0000 7f478b7fe700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 0x7f4778032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:55:57.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.442+0000 7f478b7fe700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 0x7f4778032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f47740096a0 tx=0x7f4774006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:55:57.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.446+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f477c014420 con 0x7f478c0f5410 2023-11-15T05:55:57.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.650+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f478c045af0 con 0x7f478c0f5410 2023-11-15T05:55:57.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.651+0000 7f4788ff9700 1 -- 172.21.15.137:0/39160987 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f477c01b740 con 0x7f478c0f5410 2023-11-15T05:55:57.653 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:55:57.653 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:55:57.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 msgr2=0x7f4778032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:57.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 0x7f4778032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f47740096a0 tx=0x7f4774006b40).stop 2023-11-15T05:55:57.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 msgr2=0x7f478c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:55:57.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f477c013510 tx=0x7f477c00b040).stop 2023-11-15T05:55:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 shutdown_connections 2023-11-15T05:55:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f477802ff60 0x7f4778032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 --2- 172.21.15.137:0/39160987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f478c0f5410 0x7f478c109450 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:55:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 >> 172.21.15.137:0/39160987 conn(0x7f478c0f0b70 msgr2=0x7f478c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:55:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 shutdown_connections 2023-11-15T05:55:57.656 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:55:57.652+0000 7f479254e700 1 -- 172.21.15.137:0/39160987 wait complete. 2023-11-15T05:55:57.656 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:55:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:56.974880+0000 mgr.smithi003.cldzke (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:55:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:55:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:55:57.652230+0000 mon.smithi003 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.137:0/39160987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:55:59.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:55:59.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:00.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 -- 172.21.15.137:0/734447517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 msgr2=0x7f80c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:00.239 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/734447517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c40f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f80b4009fb0 tx=0x7f80b4009b30).stop 2023-11-15T05:56:00.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 -- 172.21.15.137:0/734447517 shutdown_connections 2023-11-15T05:56:00.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/734447517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c40f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:00.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 -- 172.21.15.137:0/734447517 >> 172.21.15.137:0/734447517 conn(0x7f80c40f1c70 msgr2=0x7f80c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:00.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 -- 172.21.15.137:0/734447517 shutdown_connections 2023-11-15T05:56:00.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.237+0000 7f80ca5dc700 1 -- 172.21.15.137:0/734447517 wait complete. 2023-11-15T05:56:00.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80ca5dc700 1 Processor -- start 2023-11-15T05:56:00.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80ca5dc700 1 -- start start 2023-11-15T05:56:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80ca5dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80ca5dc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c4106c20 con 0x7f80c40f6510 2023-11-15T05:56:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80c3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80c3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48512/0 (socket says 172.21.15.137:48512) 2023-11-15T05:56:00.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80c3fff700 1 -- 172.21.15.137:0/396200456 learned_addr learned my addr 172.21.15.137:0/396200456 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:00.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.238+0000 7f80c3fff700 1 -- 172.21.15.137:0/396200456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4005040 con 0x7f80c40f6510 2023-11-15T05:56:00.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80c3fff700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f80b4013510 tx=0x7f80b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:00.244 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b4009320 con 0x7f80c40f6510 2023-11-15T05:56:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c4106e20 con 0x7f80c40f6510 2023-11-15T05:56:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80b4013f00 con 0x7f80c40f6510 2023-11-15T05:56:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b4014d90 con 0x7f80c40f6510 2023-11-15T05:56:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.239+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41072b0 con 0x7f80c40f6510 2023-11-15T05:56:00.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80b40095f0 con 0x7f80c40f6510 2023-11-15T05:56:00.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c4045ad0 con 0x7f80c40f6510 2023-11-15T05:56:00.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80c0ff9700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 0x7f80ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:00.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80b4041000 con 0x7f80c40f6510 2023-11-15T05:56:00.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80c37fe700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 0x7f80ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:00.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.240+0000 7f80c37fe700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 0x7f80ac032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f80b80096a0 tx=0x7f80b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:00.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.245+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80b4014420 con 0x7f80c40f6510 2023-11-15T05:56:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:55:58.975220+0000 mgr.smithi003.cldzke (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:00.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.449+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80c40fbf60 con 0x7f80c40f6510 2023-11-15T05:56:00.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.450+0000 7f80c0ff9700 1 -- 172.21.15.137:0/396200456 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f80b401d7f0 con 0x7f80c40f6510 2023-11-15T05:56:00.452 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:00.452 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:00.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.451+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 msgr2=0x7f80ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:00.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.451+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 0x7f80ac032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f80b80096a0 tx=0x7f80b8006b40).stop 2023-11-15T05:56:00.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.451+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 msgr2=0x7f80c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:00.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.451+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f80b4013510 tx=0x7f80b400b040).stop 2023-11-15T05:56:00.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 shutdown_connections 2023-11-15T05:56:00.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f80ac02ff60 0x7f80ac032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:00.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 --2- 172.21.15.137:0/396200456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80c40f6510 0x7f80c4106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:00.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 >> 172.21.15.137:0/396200456 conn(0x7f80c40f1c70 msgr2=0x7f80c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:00.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 shutdown_connections 2023-11-15T05:56:00.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:00.452+0000 7f80ca5dc700 1 -- 172.21.15.137:0/396200456 wait complete. 2023-11-15T05:56:00.456 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:01.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:00.451094+0000 mon.smithi003 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.137:0/396200456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:02.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:00.975581+0000 mgr.smithi003.cldzke (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:02.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:02.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:03.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 -- 172.21.15.137:0/2726172308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 msgr2=0x7f267c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:03.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 --2- 172.21.15.137:0/2726172308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c0f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f266c009fb0 tx=0x7f266c009b30).stop 2023-11-15T05:56:03.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 -- 172.21.15.137:0/2726172308 shutdown_connections 2023-11-15T05:56:03.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 --2- 172.21.15.137:0/2726172308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c0f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:03.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 -- 172.21.15.137:0/2726172308 >> 172.21.15.137:0/2726172308 conn(0x7f267c0f0b70 msgr2=0x7f267c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:03.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 -- 172.21.15.137:0/2726172308 shutdown_connections 2023-11-15T05:56:03.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.729+0000 7f26825c9700 1 -- 172.21.15.137:0/2726172308 wait complete. 2023-11-15T05:56:03.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.730+0000 7f26825c9700 1 Processor -- start 2023-11-15T05:56:03.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.730+0000 7f26825c9700 1 -- start start 2023-11-15T05:56:03.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.730+0000 7f26825c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:03.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.730+0000 7f26825c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c108dc0 con 0x7f267c060280 2023-11-15T05:56:03.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f267bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:03.736 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f267bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48518/0 (socket says 172.21.15.137:48518) 2023-11-15T05:56:03.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f267bfff700 1 -- 172.21.15.137:0/3986331173 learned_addr learned my addr 172.21.15.137:0/3986331173 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:03.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f267bfff700 1 -- 172.21.15.137:0/3986331173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c005040 con 0x7f267c060280 2023-11-15T05:56:03.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f267bfff700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f266c013510 tx=0x7f266c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:03.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c009320 con 0x7f267c060280 2023-11-15T05:56:03.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c108fc0 con 0x7f267c060280 2023-11-15T05:56:03.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c109450 con 0x7f267c060280 2023-11-15T05:56:03.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f266c013f00 con 0x7f267c060280 2023-11-15T05:56:03.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.731+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f266c014d90 con 0x7f267c060280 2023-11-15T05:56:03.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.732+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f266c0095f0 con 0x7f267c060280 2023-11-15T05:56:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.732+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c045ad0 con 0x7f267c060280 2023-11-15T05:56:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.732+0000 7f2678ff9700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 0x7f2664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.732+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f266c0410e0 con 0x7f267c060280 2023-11-15T05:56:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.733+0000 7f267b7fe700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 0x7f2664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:03.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.733+0000 7f267b7fe700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 0x7f2664032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f26700096a0 tx=0x7f2670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:03.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.737+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f266c014420 con 0x7f267c060280 2023-11-15T05:56:03.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.944+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f267c10a300 con 0x7f267c060280 2023-11-15T05:56:03.946 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.944+0000 7f2678ff9700 1 -- 172.21.15.137:0/3986331173 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f266c012980 con 0x7f267c060280 2023-11-15T05:56:03.946 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:03.947 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:03.947 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 msgr2=0x7f2664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:03.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 0x7f2664032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f26700096a0 tx=0x7f2670006b40).stop 2023-11-15T05:56:03.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 msgr2=0x7f267c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:03.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f266c013510 tx=0x7f266c00b040).stop 2023-11-15T05:56:03.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 shutdown_connections 2023-11-15T05:56:03.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f266402ff60 0x7f2664032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 --2- 172.21.15.137:0/3986331173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f267c060280 0x7f267c1088d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 >> 172.21.15.137:0/3986331173 conn(0x7f267c0f0b70 msgr2=0x7f267c0f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 shutdown_connections 2023-11-15T05:56:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:03.946+0000 7f26825c9700 1 -- 172.21.15.137:0/3986331173 wait complete. 2023-11-15T05:56:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:04.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:02.975929+0000 mgr.smithi003.cldzke (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:04.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:03.945986+0000 mon.smithi003 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.137:0/3986331173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:05.347 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:05.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:04.976207+0000 mgr.smithi003.cldzke (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:06.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 -- 172.21.15.137:0/3175001457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 msgr2=0x7fa6fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:06.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 --2- 172.21.15.137:0/3175001457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa6ec009fb0 tx=0x7fa6ec009b30).stop 2023-11-15T05:56:06.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 -- 172.21.15.137:0/3175001457 shutdown_connections 2023-11-15T05:56:06.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 --2- 172.21.15.137:0/3175001457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:06.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 -- 172.21.15.137:0/3175001457 >> 172.21.15.137:0/3175001457 conn(0x7fa6fc0f0b70 msgr2=0x7fa6fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:06.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 -- 172.21.15.137:0/3175001457 shutdown_connections 2023-11-15T05:56:06.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.511+0000 7fa702479700 1 -- 172.21.15.137:0/3175001457 wait complete. 2023-11-15T05:56:06.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.512+0000 7fa702479700 1 Processor -- start 2023-11-15T05:56:06.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.512+0000 7fa702479700 1 -- start start 2023-11-15T05:56:06.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.512+0000 7fa702479700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:06.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa702479700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc060280 con 0x7fa6fc0f5410 2023-11-15T05:56:06.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa6fbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:06.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa6fbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48534/0 (socket says 172.21.15.137:48534) 2023-11-15T05:56:06.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa6fbfff700 1 -- 172.21.15.137:0/1233786031 learned_addr learned my addr 172.21.15.137:0/1233786031 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:06.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa6fbfff700 1 -- 172.21.15.137:0/1233786031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec005040 con 0x7fa6fc0f5410 2023-11-15T05:56:06.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.513+0000 7fa6fbfff700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa6ec013510 tx=0x7fa6ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:06.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.514+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ec009320 con 0x7fa6fc0f5410 2023-11-15T05:56:06.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.514+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6fc060480 con 0x7fa6fc0f5410 2023-11-15T05:56:06.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.514+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6ec013f00 con 0x7fa6fc0f5410 2023-11-15T05:56:06.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.514+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ec014d90 con 0x7fa6fc0f5410 2023-11-15T05:56:06.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.514+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc109b20 con 0x7fa6fc0f5410 2023-11-15T05:56:06.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6ec0095f0 con 0x7fa6fc0f5410 2023-11-15T05:56:06.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6fc045ad0 con 0x7fa6fc0f5410 2023-11-15T05:56:06.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa6f8ff9700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 0x7fa6e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:06.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6ec040fe0 con 0x7fa6fc0f5410 2023-11-15T05:56:06.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa6fb7fe700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 0x7fa6e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:06.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.515+0000 7fa6fb7fe700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 0x7fa6e8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:06.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.520+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6ec01d7f0 con 0x7fa6fc0f5410 2023-11-15T05:56:06.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.726+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6fc10a340 con 0x7fa6fc0f5410 2023-11-15T05:56:06.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.727+0000 7fa6f8ff9700 1 -- 172.21.15.137:0/1233786031 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa6ec010230 con 0x7fa6fc0f5410 2023-11-15T05:56:06.729 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:06.729 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:06.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 msgr2=0x7fa6e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:06.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 0x7fa6e8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).stop 2023-11-15T05:56:06.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 msgr2=0x7fa6fc1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:06.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa6ec013510 tx=0x7fa6ec00b040).stop 2023-11-15T05:56:06.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 shutdown_connections 2023-11-15T05:56:06.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa6e802ff60 0x7fa6e8032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:06.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 --2- 172.21.15.137:0/1233786031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6fc0f5410 0x7fa6fc1093f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:06.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.728+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 >> 172.21.15.137:0/1233786031 conn(0x7fa6fc0f0b70 msgr2=0x7fa6fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:06.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.729+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 shutdown_connections 2023-11-15T05:56:06.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:06.729+0000 7fa702479700 1 -- 172.21.15.137:0/1233786031 wait complete. 2023-11-15T05:56:06.732 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:07.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:07 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:06.728398+0000 mon.smithi003 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.137:0/1233786031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:08.140 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:08.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:06.976473+0000 mgr.smithi003.cldzke (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:09.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 -- 172.21.15.137:0/2513976063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 msgr2=0x7fbeb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:09.341 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 --2- 172.21.15.137:0/2513976063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb40f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbea8009fb0 tx=0x7fbea8009b30).stop 2023-11-15T05:56:09.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 -- 172.21.15.137:0/2513976063 shutdown_connections 2023-11-15T05:56:09.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 --2- 172.21.15.137:0/2513976063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb40f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:09.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 -- 172.21.15.137:0/2513976063 >> 172.21.15.137:0/2513976063 conn(0x7fbeb40f0b70 msgr2=0x7fbeb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:09.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 -- 172.21.15.137:0/2513976063 shutdown_connections 2023-11-15T05:56:09.342 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 -- 172.21.15.137:0/2513976063 wait complete. 2023-11-15T05:56:09.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.336+0000 7fbebb133700 1 Processor -- start 2023-11-15T05:56:09.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbebb133700 1 -- start start 2023-11-15T05:56:09.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbebb133700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:09.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbebb133700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb4105b20 con 0x7fbeb40f5410 2023-11-15T05:56:09.343 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbeb8ecf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:09.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbeb8ecf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56384/0 (socket says 172.21.15.137:56384) 2023-11-15T05:56:09.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.337+0000 7fbeb8ecf700 1 -- 172.21.15.137:0/1732804965 learned_addr learned my addr 172.21.15.137:0/1732804965 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:09.344 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbeb8ecf700 1 -- 172.21.15.137:0/1732804965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8005040 con 0x7fbeb40f5410 2023-11-15T05:56:09.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbeb8ecf700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbea8013510 tx=0x7fbea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:09.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbea8009320 con 0x7fbeb40f5410 2023-11-15T05:56:09.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb4105d20 con 0x7fbeb40f5410 2023-11-15T05:56:09.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbea8013f00 con 0x7fbeb40f5410 2023-11-15T05:56:09.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbea8014d90 con 0x7fbeb40f5410 2023-11-15T05:56:09.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.338+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb41061b0 con 0x7fbeb40f5410 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbea80263a0 con 0x7fbeb40f5410 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb4045ad0 con 0x7fbeb40f5410 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbeb17fa700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbea80410c0 con 0x7fbeb40f5410 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbeb3fff700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:09.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.339+0000 7fbeb3fff700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 0x7fbea4032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:09.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.343+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbea80095f0 con 0x7fbeb40f5410 2023-11-15T05:56:09.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.548+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbeb40fae60 con 0x7fbeb40f5410 2023-11-15T05:56:09.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.548+0000 7fbeb17fa700 1 -- 172.21.15.137:0/1732804965 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbea801db70 con 0x7fbeb40f5410 2023-11-15T05:56:09.551 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:09.551 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:09.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 msgr2=0x7fbea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:09.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 0x7fbea4032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).stop 2023-11-15T05:56:09.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 msgr2=0x7fbeb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:09.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fbea8013510 tx=0x7fbea800b040).stop 2023-11-15T05:56:09.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 shutdown_connections 2023-11-15T05:56:09.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbea402ff60 0x7fbea4032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:09.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 --2- 172.21.15.137:0/1732804965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbeb40f5410 0x7fbeb4105630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:09.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 >> 172.21.15.137:0/1732804965 conn(0x7fbeb40f0b70 msgr2=0x7fbeb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:09.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 shutdown_connections 2023-11-15T05:56:09.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:09.550+0000 7fbebb133700 1 -- 172.21.15.137:0/1732804965 wait complete. 2023-11-15T05:56:09.555 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:08.976784+0000 mgr.smithi003.cldzke (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:09.549933+0000 mon.smithi003 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.137:0/1732804965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:10.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:10.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:10.977116+0000 mgr.smithi003.cldzke (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:12.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 -- 172.21.15.137:0/270810997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 msgr2=0x7f2bd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:12.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/270810997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd80f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2bcc009fb0 tx=0x7f2bcc009b30).stop 2023-11-15T05:56:12.174 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 -- 172.21.15.137:0/270810997 shutdown_connections 2023-11-15T05:56:12.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/270810997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd80f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:12.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 -- 172.21.15.137:0/270810997 >> 172.21.15.137:0/270810997 conn(0x7f2bd80f1c70 msgr2=0x7f2bd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:12.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 -- 172.21.15.137:0/270810997 shutdown_connections 2023-11-15T05:56:12.175 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.172+0000 7f2bdcd69700 1 -- 172.21.15.137:0/270810997 wait complete. 2023-11-15T05:56:12.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bdcd69700 1 Processor -- start 2023-11-15T05:56:12.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bdcd69700 1 -- start start 2023-11-15T05:56:12.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bdcd69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:12.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bdcd69700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd8106c20 con 0x7f2bd80f6510 2023-11-15T05:56:12.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:12.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56396/0 (socket says 172.21.15.137:56396) 2023-11-15T05:56:12.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.173+0000 7f2bd659c700 1 -- 172.21.15.137:0/281175481 learned_addr learned my addr 172.21.15.137:0/281175481 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:12.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bd659c700 1 -- 172.21.15.137:0/281175481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc005040 con 0x7f2bd80f6510 2023-11-15T05:56:12.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bd659c700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2bcc013510 tx=0x7f2bcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:12.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc009320 con 0x7f2bd80f6510 2023-11-15T05:56:12.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd8106e20 con 0x7f2bd80f6510 2023-11-15T05:56:12.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bcc013f00 con 0x7f2bd80f6510 2023-11-15T05:56:12.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bcc014d90 con 0x7f2bd80f6510 2023-11-15T05:56:12.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.174+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd81072b0 con 0x7f2bd80f6510 2023-11-15T05:56:12.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.175+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bcc0095f0 con 0x7f2bd80f6510 2023-11-15T05:56:12.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.175+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd8045ad0 con 0x7f2bd80f6510 2023-11-15T05:56:12.181 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.175+0000 7f2bc6ffd700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 0x7f2bc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:12.182 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.175+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bcc0410a0 con 0x7f2bd80f6510 2023-11-15T05:56:12.182 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.175+0000 7f2bd5d9b700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 0x7f2bc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:12.182 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.176+0000 7f2bd5d9b700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 0x7f2bc0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2bc80096a0 tx=0x7f2bc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:12.182 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.180+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bcc010230 con 0x7f2bd80f6510 2023-11-15T05:56:12.387 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.385+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bd80fbf60 con 0x7f2bd80f6510 2023-11-15T05:56:12.388 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.386+0000 7f2bc6ffd700 1 -- 172.21.15.137:0/281175481 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2bcc010230 con 0x7f2bd80f6510 2023-11-15T05:56:12.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:12.388 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:12.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 msgr2=0x7f2bc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:12.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 0x7f2bc0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2bc80096a0 tx=0x7f2bc8006b40).stop 2023-11-15T05:56:12.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 msgr2=0x7f2bd8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:12.390 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2bcc013510 tx=0x7f2bcc00b040).stop 2023-11-15T05:56:12.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 shutdown_connections 2023-11-15T05:56:12.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2bc002ff60 0x7f2bc0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:12.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 --2- 172.21.15.137:0/281175481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd80f6510 0x7f2bd8106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:12.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 >> 172.21.15.137:0/281175481 conn(0x7f2bd80f1c70 msgr2=0x7f2bd80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:12.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 shutdown_connections 2023-11-15T05:56:12.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:12.388+0000 7f2bdcd69700 1 -- 172.21.15.137:0/281175481 wait complete. 2023-11-15T05:56:12.392 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:12.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:12.387618+0000 mon.smithi003 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.137:0/281175481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:13.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:13.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:13.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:12.977461+0000 mgr.smithi003.cldzke (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:14.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 -- 172.21.15.137:0/1200617129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 msgr2=0x7f62040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:14.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 --2- 172.21.15.137:0/1200617129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f62040f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f61f8009fb0 tx=0x7f61f8009b30).stop 2023-11-15T05:56:14.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 -- 172.21.15.137:0/1200617129 shutdown_connections 2023-11-15T05:56:14.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 --2- 172.21.15.137:0/1200617129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f62040f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 -- 172.21.15.137:0/1200617129 >> 172.21.15.137:0/1200617129 conn(0x7f62040f1c70 msgr2=0x7f62040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 -- 172.21.15.137:0/1200617129 shutdown_connections 2023-11-15T05:56:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 -- 172.21.15.137:0/1200617129 wait complete. 2023-11-15T05:56:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.961+0000 7f620b051700 1 Processor -- start 2023-11-15T05:56:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f620b051700 1 -- start start 2023-11-15T05:56:14.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f620b051700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:14.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f620b051700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6204069e20 con 0x7f62040f6510 2023-11-15T05:56:14.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f6208ded700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:14.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f6208ded700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56422/0 (socket says 172.21.15.137:56422) 2023-11-15T05:56:14.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f6208ded700 1 -- 172.21.15.137:0/1793826274 learned_addr learned my addr 172.21.15.137:0/1793826274 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:14.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f6208ded700 1 -- 172.21.15.137:0/1793826274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8005040 con 0x7f62040f6510 2023-11-15T05:56:14.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.962+0000 7f6208ded700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f61f8013510 tx=0x7f61f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:14.968 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f8009320 con 0x7f62040f6510 2023-11-15T05:56:14.969 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6204069fc0 con 0x7f62040f6510 2023-11-15T05:56:14.969 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61f8013f00 con 0x7f62040f6510 2023-11-15T05:56:14.969 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61f8014d90 con 0x7f62040f6510 2023-11-15T05:56:14.969 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620406a450 con 0x7f62040f6510 2023-11-15T05:56:14.970 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.963+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61f80263a0 con 0x7f62040f6510 2023-11-15T05:56:14.970 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.964+0000 7f62017fa700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 0x7f61f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:14.970 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.964+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61f8041040 con 0x7f62040f6510 2023-11-15T05:56:14.970 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.964+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6204059100 con 0x7f62040f6510 2023-11-15T05:56:14.971 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.964+0000 7f6203fff700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 0x7f61f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:14.971 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.964+0000 7f6203fff700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 0x7f61f4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f61f00096a0 tx=0x7f61f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:14.971 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:14.969+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61f8012920 con 0x7f62040f6510 2023-11-15T05:56:15.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.174+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6204045ad0 con 0x7f62040f6510 2023-11-15T05:56:15.176 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.175+0000 7f62017fa700 1 -- 172.21.15.137:0/1793826274 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f61f802b020 con 0x7f62040f6510 2023-11-15T05:56:15.177 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:15.177 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:15.178 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 msgr2=0x7f61f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:15.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 0x7f61f4032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f61f00096a0 tx=0x7f61f0006b40).stop 2023-11-15T05:56:15.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 msgr2=0x7f620406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:15.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f61f8013510 tx=0x7f61f800b040).stop 2023-11-15T05:56:15.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 shutdown_connections 2023-11-15T05:56:15.179 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f61f402ff60 0x7f61f4032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:15.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 --2- 172.21.15.137:0/1793826274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62040f6510 0x7f620406b9a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:15.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 >> 172.21.15.137:0/1793826274 conn(0x7f62040f1c70 msgr2=0x7f62040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:15.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 shutdown_connections 2023-11-15T05:56:15.180 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:15.177+0000 7f620b051700 1 -- 172.21.15.137:0/1793826274 wait complete. 2023-11-15T05:56:15.181 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:16.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:14.977772+0000 mgr.smithi003.cldzke (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:15.176581+0000 mon.smithi003 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.137:0/1793826274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:16.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:16.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:17.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 -- 172.21.15.137:0/2799945385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 msgr2=0x7f324c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:17.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 --2- 172.21.15.137:0/2799945385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c0f9150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3234009fb0 tx=0x7f3234009b30).stop 2023-11-15T05:56:17.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 -- 172.21.15.137:0/2799945385 shutdown_connections 2023-11-15T05:56:17.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 --2- 172.21.15.137:0/2799945385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c0f9150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:17.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 -- 172.21.15.137:0/2799945385 >> 172.21.15.137:0/2799945385 conn(0x7f324c0f0b70 msgr2=0x7f324c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:17.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 -- 172.21.15.137:0/2799945385 shutdown_connections 2023-11-15T05:56:17.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.794+0000 7f3252315700 1 -- 172.21.15.137:0/2799945385 wait complete. 2023-11-15T05:56:17.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.795+0000 7f3252315700 1 Processor -- start 2023-11-15T05:56:17.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.795+0000 7f3252315700 1 -- start start 2023-11-15T05:56:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.795+0000 7f3252315700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.795+0000 7f3252315700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c106bd0 con 0x7f324c060220 2023-11-15T05:56:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f324bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f324bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56432/0 (socket says 172.21.15.137:56432) 2023-11-15T05:56:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f324bfff700 1 -- 172.21.15.137:0/441024708 learned_addr learned my addr 172.21.15.137:0/441024708 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f324bfff700 1 -- 172.21.15.137:0/441024708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234005040 con 0x7f324c060220 2023-11-15T05:56:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f324bfff700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3234013510 tx=0x7f323400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3234009320 con 0x7f324c060220 2023-11-15T05:56:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c106dd0 con 0x7f324c060220 2023-11-15T05:56:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3234013f00 con 0x7f324c060220 2023-11-15T05:56:17.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3234014dc0 con 0x7f324c060220 2023-11-15T05:56:17.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.796+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c107260 con 0x7f324c060220 2023-11-15T05:56:17.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.797+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f323401d3d0 con 0x7f324c060220 2023-11-15T05:56:17.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.797+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c045ad0 con 0x7f324c060220 2023-11-15T05:56:17.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.797+0000 7f3248ff9700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:17.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.797+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32340413c0 con 0x7f324c060220 2023-11-15T05:56:17.802 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.798+0000 7f324b7fe700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:17.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.798+0000 7f324b7fe700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 0x7f3238032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f323c0096a0 tx=0x7f323c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:17.803 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:17.802+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f323401ccb0 con 0x7f324c060220 2023-11-15T05:56:18.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.007+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f324c108110 con 0x7f324c060220 2023-11-15T05:56:18.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.008+0000 7f3248ff9700 1 -- 172.21.15.137:0/441024708 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f323401cad0 con 0x7f324c060220 2023-11-15T05:56:18.010 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:18.010 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:18.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 msgr2=0x7f3238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:18.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 0x7f3238032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f323c0096a0 tx=0x7f323c006b40).stop 2023-11-15T05:56:18.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 msgr2=0x7f324c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:18.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3234013510 tx=0x7f323400b040).stop 2023-11-15T05:56:18.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 shutdown_connections 2023-11-15T05:56:18.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f323802ff60 0x7f3238032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:18.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 --2- 172.21.15.137:0/441024708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f324c060220 0x7f324c1066e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:18.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 >> 172.21.15.137:0/441024708 conn(0x7f324c0f0b70 msgr2=0x7f324c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:18.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 shutdown_connections 2023-11-15T05:56:18.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:18.010+0000 7f3252315700 1 -- 172.21.15.137:0/441024708 wait complete. 2023-11-15T05:56:18.013 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:18.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:16.978117+0000 mgr.smithi003.cldzke (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:18.008416+0000 mon.smithi003 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.137:0/441024708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:19.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:19.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:20.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:18.978412+0000 mgr.smithi003.cldzke (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:20.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110152032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 msgr2=0x7f2b740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:20.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110152032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2b64009960 tx=0x7f2b64004d10).stop 2023-11-15T05:56:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110152032 shutdown_connections 2023-11-15T05:56:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110152032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110152032 >> 172.21.15.137:0/110152032 conn(0x7f2b740f0b70 msgr2=0x7f2b740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110152032 shutdown_connections 2023-11-15T05:56:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.601+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110152032 wait complete. 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b7beb0700 1 Processor -- start 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b7beb0700 1 -- start start 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b7beb0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b7beb0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740fb820 con 0x7f2b740f5410 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b79c4c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:20.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b79c4c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44986/0 (socket says 172.21.15.137:44986) 2023-11-15T05:56:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b79c4c700 1 -- 172.21.15.137:0/110215518 learned_addr learned my addr 172.21.15.137:0/110215518 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b79c4c700 1 -- 172.21.15.137:0/110215518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009640 con 0x7f2b740f5410 2023-11-15T05:56:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.602+0000 7f2b79c4c700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2b64004b70 tx=0x7f2b64004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b64018070 con 0x7f2b740f5410 2023-11-15T05:56:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b740fb9c0 con 0x7f2b740f5410 2023-11-15T05:56:20.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b740fbe50 con 0x7f2b740f5410 2023-11-15T05:56:20.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b6400b870 con 0x7f2b740f5410 2023-11-15T05:56:20.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b6401c560 con 0x7f2b740f5410 2023-11-15T05:56:20.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.603+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b6401c6c0 con 0x7f2b740f5410 2023-11-15T05:56:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.604+0000 7f2b727fc700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.604+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b54004fa0 con 0x7f2b740f5410 2023-11-15T05:56:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.604+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b640412b0 con 0x7f2b740f5410 2023-11-15T05:56:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.604+0000 7f2b7944b700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.604+0000 7f2b7944b700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 0x7f2b5c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f2b680096a0 tx=0x7f2b68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:20.610 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.609+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b64012be0 con 0x7f2b740f5410 2023-11-15T05:56:20.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.814+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b54005670 con 0x7f2b740f5410 2023-11-15T05:56:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.815+0000 7f2b727fc700 1 -- 172.21.15.137:0/110215518 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2b64004e10 con 0x7f2b740f5410 2023-11-15T05:56:20.816 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:20.816 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:20.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 msgr2=0x7f2b5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:20.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 0x7f2b5c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f2b680096a0 tx=0x7f2b68006b40).stop 2023-11-15T05:56:20.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 msgr2=0x7f2b740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:20.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2b64004b70 tx=0x7f2b64004d10).stop 2023-11-15T05:56:20.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 shutdown_connections 2023-11-15T05:56:20.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:20.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 --2- 172.21.15.137:0/110215518 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2b740f5410 0x7f2b740fd3a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:20.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 >> 172.21.15.137:0/110215518 conn(0x7f2b740f0b70 msgr2=0x7f2b7405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:20.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 shutdown_connections 2023-11-15T05:56:20.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:20.816+0000 7f2b7beb0700 1 -- 172.21.15.137:0/110215518 wait complete. 2023-11-15T05:56:20.819 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:21.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:20.814929+0000 mon.smithi003 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.137:0/110215518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:20.978781+0000 mgr.smithi003.cldzke (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:22.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:22.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:23.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- 172.21.15.137:0/219848897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 msgr2=0x7f0c500f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:23.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 --2- 172.21.15.137:0/219848897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c500f6bc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0c4c009fb0 tx=0x7f0c4c009b30).stop 2023-11-15T05:56:23.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- 172.21.15.137:0/219848897 shutdown_connections 2023-11-15T05:56:23.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 --2- 172.21.15.137:0/219848897 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c500f6bc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:23.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- 172.21.15.137:0/219848897 >> 172.21.15.137:0/219848897 conn(0x7f0c500f0120 msgr2=0x7f0c500f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:23.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- 172.21.15.137:0/219848897 shutdown_connections 2023-11-15T05:56:23.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- 172.21.15.137:0/219848897 wait complete. 2023-11-15T05:56:23.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 Processor -- start 2023-11-15T05:56:23.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.427+0000 7f0c574a6700 1 -- start start 2023-11-15T05:56:23.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c574a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:23.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c574a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c501058c0 con 0x7f0c500f67a0 2023-11-15T05:56:23.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c564a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:23.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c564a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44994/0 (socket says 172.21.15.137:44994) 2023-11-15T05:56:23.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c564a4700 1 -- 172.21.15.137:0/682530104 learned_addr learned my addr 172.21.15.137:0/682530104 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:23.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c564a4700 1 -- 172.21.15.137:0/682530104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c005040 con 0x7f0c500f67a0 2023-11-15T05:56:23.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c564a4700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f0c4c013510 tx=0x7f0c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:23.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c4c009320 con 0x7f0c500f67a0 2023-11-15T05:56:23.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c50105ac0 con 0x7f0c500f67a0 2023-11-15T05:56:23.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c4c013f00 con 0x7f0c500f67a0 2023-11-15T05:56:23.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c4c014dc0 con 0x7f0c500f67a0 2023-11-15T05:56:23.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.428+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c50105f50 con 0x7f0c500f67a0 2023-11-15T05:56:23.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.429+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c4c0095f0 con 0x7f0c500f67a0 2023-11-15T05:56:23.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.429+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c50045ad0 con 0x7f0c500f67a0 2023-11-15T05:56:23.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.429+0000 7f0c46ffd700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 0x7f0c3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:23.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.429+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c4c041100 con 0x7f0c500f67a0 2023-11-15T05:56:23.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.430+0000 7f0c55ca3700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 0x7f0c3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:23.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.430+0000 7f0c55ca3700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 0x7f0c3c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0c400096a0 tx=0x7f0c40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:23.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.434+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c4c0268c0 con 0x7f0c500f67a0 2023-11-15T05:56:23.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.643+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c50106e00 con 0x7f0c500f67a0 2023-11-15T05:56:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.644+0000 7f0c46ffd700 1 -- 172.21.15.137:0/682530104 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0c4c010230 con 0x7f0c500f67a0 2023-11-15T05:56:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:23.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 msgr2=0x7f0c3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 0x7f0c3c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0c400096a0 tx=0x7f0c40006b40).stop 2023-11-15T05:56:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 msgr2=0x7f0c501053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f0c4c013510 tx=0x7f0c4c00b040).stop 2023-11-15T05:56:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 shutdown_connections 2023-11-15T05:56:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3c02ff60 0x7f0c3c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 --2- 172.21.15.137:0/682530104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c500f67a0 0x7f0c501053d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 >> 172.21.15.137:0/682530104 conn(0x7f0c500f0120 msgr2=0x7f0c500f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 shutdown_connections 2023-11-15T05:56:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:23.645+0000 7f0c574a6700 1 -- 172.21.15.137:0/682530104 wait complete. 2023-11-15T05:56:23.648 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:22.979140+0000 mgr.smithi003.cldzke (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:23.644029+0000 mon.smithi003 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.137:0/682530104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:25.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:25.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:26.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 -- 172.21.15.137:0/460483205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 msgr2=0x7f2d7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 --2- 172.21.15.137:0/460483205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c0f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2d6c009fb0 tx=0x7f2d6c009b30).stop 2023-11-15T05:56:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 -- 172.21.15.137:0/460483205 shutdown_connections 2023-11-15T05:56:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 --2- 172.21.15.137:0/460483205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c0f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:26.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 -- 172.21.15.137:0/460483205 >> 172.21.15.137:0/460483205 conn(0x7f2d7c0f1c70 msgr2=0x7f2d7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:26.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 -- 172.21.15.137:0/460483205 shutdown_connections 2023-11-15T05:56:26.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 -- 172.21.15.137:0/460483205 wait complete. 2023-11-15T05:56:26.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.254+0000 7f2d823f2700 1 Processor -- start 2023-11-15T05:56:26.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d823f2700 1 -- start start 2023-11-15T05:56:26.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d823f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:26.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d823f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c069e20 con 0x7f2d7c0f6510 2023-11-15T05:56:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d7bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45014/0 (socket says 172.21.15.137:45014) 2023-11-15T05:56:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d7bfff700 1 -- 172.21.15.137:0/1506244874 learned_addr learned my addr 172.21.15.137:0/1506244874 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d7bfff700 1 -- 172.21.15.137:0/1506244874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c005040 con 0x7f2d7c0f6510 2023-11-15T05:56:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.255+0000 7f2d7bfff700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2d6c013510 tx=0x7f2d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:26.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d6c009320 con 0x7f2d7c0f6510 2023-11-15T05:56:26.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c069fc0 con 0x7f2d7c0f6510 2023-11-15T05:56:26.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d6c013f00 con 0x7f2d7c0f6510 2023-11-15T05:56:26.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d6c014d90 con 0x7f2d7c0f6510 2023-11-15T05:56:26.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c06a450 con 0x7f2d7c0f6510 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d6c014420 con 0x7f2d7c0f6510 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c059100 con 0x7f2d7c0f6510 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.256+0000 7f2d78ff9700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 0x7f2d64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.257+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d6c041070 con 0x7f2d7c0f6510 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.257+0000 7f2d7b7fe700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 0x7f2d64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:26.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.257+0000 7f2d7b7fe700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 0x7f2d64032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d700096a0 tx=0x7f2d70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:26.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.261+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d6c0095f0 con 0x7f2d7c0f6510 2023-11-15T05:56:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:24.979399+0000 mgr.smithi003.cldzke (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:26.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.466+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d7c0fbf60 con 0x7f2d7c0f6510 2023-11-15T05:56:26.467 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.467+0000 7f2d78ff9700 1 -- 172.21.15.137:0/1506244874 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2d6c01db70 con 0x7f2d7c0f6510 2023-11-15T05:56:26.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:26.468 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:26.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 msgr2=0x7f2d64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:26.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 0x7f2d64032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d700096a0 tx=0x7f2d70006b40).stop 2023-11-15T05:56:26.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 msgr2=0x7f2d7c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:26.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f2d6c013510 tx=0x7f2d6c00b040).stop 2023-11-15T05:56:26.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 shutdown_connections 2023-11-15T05:56:26.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d6402ff60 0x7f2d64032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:26.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 --2- 172.21.15.137:0/1506244874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d7c0f6510 0x7f2d7c06b9a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:26.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 >> 172.21.15.137:0/1506244874 conn(0x7f2d7c0f1c70 msgr2=0x7f2d7c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:26.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 shutdown_connections 2023-11-15T05:56:26.471 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:26.468+0000 7f2d823f2700 1 -- 172.21.15.137:0/1506244874 wait complete. 2023-11-15T05:56:26.471 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:27.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:27 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:26.467115+0000 mon.smithi003 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.137:0/1506244874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:27.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:27.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:28.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:26.979765+0000 mgr.smithi003.cldzke (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:29.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 -- 172.21.15.137:0/1540192354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 msgr2=0x7f06b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:29.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 --2- 172.21.15.137:0/1540192354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b80f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f06ac009fb0 tx=0x7f06ac009b30).stop 2023-11-15T05:56:29.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 -- 172.21.15.137:0/1540192354 shutdown_connections 2023-11-15T05:56:29.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 --2- 172.21.15.137:0/1540192354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b80f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:29.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 -- 172.21.15.137:0/1540192354 >> 172.21.15.137:0/1540192354 conn(0x7f06b80f0b70 msgr2=0x7f06b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 -- 172.21.15.137:0/1540192354 shutdown_connections 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.060+0000 7f06bf417700 1 -- 172.21.15.137:0/1540192354 wait complete. 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bf417700 1 Processor -- start 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bf417700 1 -- start start 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bf417700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:29.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bf417700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8105b20 con 0x7f06b80f5410 2023-11-15T05:56:29.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bd1b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:29.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bd1b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45038/0 (socket says 172.21.15.137:45038) 2023-11-15T05:56:29.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bd1b3700 1 -- 172.21.15.137:0/2905254917 learned_addr learned my addr 172.21.15.137:0/2905254917 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:29.065 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bd1b3700 1 -- 172.21.15.137:0/2905254917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac005040 con 0x7f06b80f5410 2023-11-15T05:56:29.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.061+0000 7f06bd1b3700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f06ac013510 tx=0x7f06ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:29.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.062+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06ac009320 con 0x7f06b80f5410 2023-11-15T05:56:29.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.062+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b8105d20 con 0x7f06b80f5410 2023-11-15T05:56:29.066 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.062+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06ac013f00 con 0x7f06b80f5410 2023-11-15T05:56:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.062+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06ac014d90 con 0x7f06b80f5410 2023-11-15T05:56:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.062+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b81061b0 con 0x7f06b80f5410 2023-11-15T05:56:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.063+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8045ad0 con 0x7f06b80f5410 2023-11-15T05:56:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.063+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06ac0095f0 con 0x7f06b80f5410 2023-11-15T05:56:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.063+0000 7f06a9ffb700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 0x7f06a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:29.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.063+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06ac041000 con 0x7f06b80f5410 2023-11-15T05:56:29.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.063+0000 7f06bc9b2700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 0x7f06a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:29.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.064+0000 7f06bc9b2700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 0x7f06a4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f06b40096a0 tx=0x7f06b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:29.068 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.068+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06ac01d7f0 con 0x7f06b80f5410 2023-11-15T05:56:29.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.273+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06b80fae60 con 0x7f06b80f5410 2023-11-15T05:56:29.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.274+0000 7f06a9ffb700 1 -- 172.21.15.137:0/2905254917 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f06ac010230 con 0x7f06b80f5410 2023-11-15T05:56:29.274 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:29.275 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:29.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 msgr2=0x7f06a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:29.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 0x7f06a4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f06b40096a0 tx=0x7f06b4006b40).stop 2023-11-15T05:56:29.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 msgr2=0x7f06b8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:29.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f06ac013510 tx=0x7f06ac00b040).stop 2023-11-15T05:56:29.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 shutdown_connections 2023-11-15T05:56:29.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f06a402ff60 0x7f06a4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:29.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 --2- 172.21.15.137:0/2905254917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06b80f5410 0x7f06b8105630 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:29.278 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.275+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 >> 172.21.15.137:0/2905254917 conn(0x7f06b80f0b70 msgr2=0x7f06b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:29.278 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.276+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 shutdown_connections 2023-11-15T05:56:29.278 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:29.276+0000 7f06bf417700 1 -- 172.21.15.137:0/2905254917 wait complete. 2023-11-15T05:56:29.278 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:28.980143+0000 mgr.smithi003.cldzke (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:29.274014+0000 mon.smithi003 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.137:0/2905254917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:30.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:30.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:31.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 -- 172.21.15.137:0/472740798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 msgr2=0x7fe7400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:31.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 --2- 172.21.15.137:0/472740798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe7400f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fe730009fb0 tx=0x7fe730009b30).stop 2023-11-15T05:56:31.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 -- 172.21.15.137:0/472740798 shutdown_connections 2023-11-15T05:56:31.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 --2- 172.21.15.137:0/472740798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe7400f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:31.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 -- 172.21.15.137:0/472740798 >> 172.21.15.137:0/472740798 conn(0x7fe7400f1c70 msgr2=0x7fe7400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:31.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 -- 172.21.15.137:0/472740798 shutdown_connections 2023-11-15T05:56:31.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.836+0000 7fe74587c700 1 -- 172.21.15.137:0/472740798 wait complete. 2023-11-15T05:56:31.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe74587c700 1 Processor -- start 2023-11-15T05:56:31.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe74587c700 1 -- start start 2023-11-15T05:56:31.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe74587c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:31.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe74587c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740106c20 con 0x7fe7400f6510 2023-11-15T05:56:31.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe73effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:31.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe73effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34158/0 (socket says 172.21.15.137:34158) 2023-11-15T05:56:31.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe73effd700 1 -- 172.21.15.137:0/878433802 learned_addr learned my addr 172.21.15.137:0/878433802 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:31.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.837+0000 7fe73effd700 1 -- 172.21.15.137:0/878433802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730005040 con 0x7fe7400f6510 2023-11-15T05:56:31.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe73effd700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe730014000 tx=0x7fe73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:31.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7300095f0 con 0x7fe7400f6510 2023-11-15T05:56:31.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe740106e20 con 0x7fe7400f6510 2023-11-15T05:56:31.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe730007e70 con 0x7fe7400f6510 2023-11-15T05:56:31.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe730014ea0 con 0x7fe7400f6510 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7401072b0 con 0x7fe7400f6510 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.838+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe730025070 con 0x7fe7400f6510 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.839+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe740045ad0 con 0x7fe7400f6510 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.839+0000 7fe7277fe700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 0x7fe728032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.839+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe730040980 con 0x7fe7400f6510 2023-11-15T05:56:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.839+0000 7fe73e7fc700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 0x7fe728032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:31.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.839+0000 7fe73e7fc700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 0x7fe728032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe7340096a0 tx=0x7fe734006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:31.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:31.844+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7300079c0 con 0x7fe7400f6510 2023-11-15T05:56:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:30.980561+0000 mgr.smithi003.cldzke (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:32.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.050+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7400fbf60 con 0x7fe7400f6510 2023-11-15T05:56:32.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.051+0000 7fe7277fe700 1 -- 172.21.15.137:0/878433802 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe73000f300 con 0x7fe7400f6510 2023-11-15T05:56:32.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:32.052 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:32.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 msgr2=0x7fe728032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:32.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 0x7fe728032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe7340096a0 tx=0x7fe734006b40).stop 2023-11-15T05:56:32.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 msgr2=0x7fe740106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:32.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe730014000 tx=0x7fe73000b040).stop 2023-11-15T05:56:32.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 shutdown_connections 2023-11-15T05:56:32.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe72802ff60 0x7fe728032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:32.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 --2- 172.21.15.137:0/878433802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7400f6510 0x7fe740106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:32.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 >> 172.21.15.137:0/878433802 conn(0x7fe7400f1c70 msgr2=0x7fe7400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:32.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 shutdown_connections 2023-11-15T05:56:32.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:32.052+0000 7fe74587c700 1 -- 172.21.15.137:0/878433802 wait complete. 2023-11-15T05:56:32.055 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:32.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:32.051158+0000 mon.smithi003 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.137:0/878433802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:33.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:33.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:33.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:33 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:32.980926+0000 mgr.smithi003.cldzke (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:34.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.677+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1903286913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 msgr2=0x7ff5980f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:34.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.677+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1903286913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5980f9150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff590009960 tx=0x7ff590004d10).stop 2023-11-15T05:56:34.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.677+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1903286913 shutdown_connections 2023-11-15T05:56:34.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.677+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1903286913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5980f9150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:34.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.677+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1903286913 >> 172.21.15.137:0/1903286913 conn(0x7ff5980f0b70 msgr2=0x7ff5980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:34.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1903286913 shutdown_connections 2023-11-15T05:56:34.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1903286913 wait complete. 2023-11-15T05:56:34.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 Processor -- start 2023-11-15T05:56:34.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 -- start start 2023-11-15T05:56:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59fbb1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff598106bd0 con 0x7ff598060220 2023-11-15T05:56:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59d94d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59d94d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34184/0 (socket says 172.21.15.137:34184) 2023-11-15T05:56:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.678+0000 7ff59d94d700 1 -- 172.21.15.137:0/1798985539 learned_addr learned my addr 172.21.15.137:0/1798985539 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:34.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff59d94d700 1 -- 172.21.15.137:0/1798985539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590009640 con 0x7ff598060220 2023-11-15T05:56:34.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff59d94d700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff590007bd0 tx=0x7ff590004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:34.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff590018070 con 0x7ff598060220 2023-11-15T05:56:34.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff598106dd0 con 0x7ff598060220 2023-11-15T05:56:34.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff59000b870 con 0x7ff598060220 2023-11-15T05:56:34.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff59001d420 con 0x7ff598060220 2023-11-15T05:56:34.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.679+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff598107260 con 0x7ff598060220 2023-11-15T05:56:34.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.680+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff590024060 con 0x7ff598060220 2023-11-15T05:56:34.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.680+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff598045ad0 con 0x7ff598060220 2023-11-15T05:56:34.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.680+0000 7ff58e7fc700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:34.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.680+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff590041200 con 0x7ff598060220 2023-11-15T05:56:34.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.680+0000 7ff59d14c700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:34.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.681+0000 7ff59d14c700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 0x7ff584032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff5880096a0 tx=0x7ff588006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:34.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.685+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff590025520 con 0x7ff598060220 2023-11-15T05:56:34.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.889+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff598108060 con 0x7ff598060220 2023-11-15T05:56:34.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.889+0000 7ff58e7fc700 1 -- 172.21.15.137:0/1798985539 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff590025340 con 0x7ff598060220 2023-11-15T05:56:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:34.891 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:34.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 msgr2=0x7ff584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:34.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 0x7ff584032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff5880096a0 tx=0x7ff588006b40).stop 2023-11-15T05:56:34.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 msgr2=0x7ff5981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:34.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff590007bd0 tx=0x7ff590004d10).stop 2023-11-15T05:56:34.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 shutdown_connections 2023-11-15T05:56:34.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff58402ff60 0x7ff584032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:34.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 --2- 172.21.15.137:0/1798985539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff598060220 0x7ff5981066e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 >> 172.21.15.137:0/1798985539 conn(0x7ff5980f0b70 msgr2=0x7ff5980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 shutdown_connections 2023-11-15T05:56:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:34.891+0000 7ff59fbb1700 1 -- 172.21.15.137:0/1798985539 wait complete. 2023-11-15T05:56:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:35.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:34.889649+0000 mon.smithi003 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.137:0/1798985539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:36.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:36.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:34.981315+0000 mgr.smithi003.cldzke (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:35.111192+0000 mon.smithi003 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:56:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:35.114575+0000 mon.smithi003 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:56:37.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.510+0000 7f091b05d700 1 -- 172.21.15.137:0/3610886381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 msgr2=0x7f09140f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:37.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.510+0000 7f091b05d700 1 --2- 172.21.15.137:0/3610886381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09140f68a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f0908009fb0 tx=0x7f0908009b30).stop 2023-11-15T05:56:37.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.510+0000 7f091b05d700 1 -- 172.21.15.137:0/3610886381 shutdown_connections 2023-11-15T05:56:37.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.510+0000 7f091b05d700 1 --2- 172.21.15.137:0/3610886381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09140f68a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:37.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.510+0000 7f091b05d700 1 -- 172.21.15.137:0/3610886381 >> 172.21.15.137:0/3610886381 conn(0x7f09140f1c00 msgr2=0x7f09140f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:37.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.511+0000 7f091b05d700 1 -- 172.21.15.137:0/3610886381 shutdown_connections 2023-11-15T05:56:37.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.511+0000 7f091b05d700 1 -- 172.21.15.137:0/3610886381 wait complete. 2023-11-15T05:56:37.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.511+0000 7f091b05d700 1 Processor -- start 2023-11-15T05:56:37.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.511+0000 7f091b05d700 1 -- start start 2023-11-15T05:56:37.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f091b05d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:37.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f091b05d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0914106bc0 con 0x7f09140f6480 2023-11-15T05:56:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f0918df9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f0918df9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34200/0 (socket says 172.21.15.137:34200) 2023-11-15T05:56:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f0918df9700 1 -- 172.21.15.137:0/4196517874 learned_addr learned my addr 172.21.15.137:0/4196517874 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.512+0000 7f0918df9700 1 -- 172.21.15.137:0/4196517874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908005040 con 0x7f09140f6480 2023-11-15T05:56:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f0918df9700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0908013510 tx=0x7f090800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:37.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0908009320 con 0x7f09140f6480 2023-11-15T05:56:37.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0914106dc0 con 0x7f09140f6480 2023-11-15T05:56:37.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0914107250 con 0x7f09140f6480 2023-11-15T05:56:37.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0908013f00 con 0x7f09140f6480 2023-11-15T05:56:37.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.513+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0908014d60 con 0x7f09140f6480 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.514+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09080263a0 con 0x7f09140f6480 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.514+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0914045ad0 con 0x7f09140f6480 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.514+0000 7f09117fa700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 0x7f0904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.514+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0908041360 con 0x7f09140f6480 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.514+0000 7f0913fff700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 0x7f0904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:37.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.515+0000 7f0913fff700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 0x7f0904032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f09000096a0 tx=0x7f0900006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:37.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.519+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0908012510 con 0x7f09140f6480 2023-11-15T05:56:37.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.725+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0914108100 con 0x7f09140f6480 2023-11-15T05:56:37.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.725+0000 7f09117fa700 1 -- 172.21.15.137:0/4196517874 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f090801ce90 con 0x7f09140f6480 2023-11-15T05:56:37.727 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:37.728 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:37.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 msgr2=0x7f0904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:37.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 0x7f0904032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f09000096a0 tx=0x7f0900006b40).stop 2023-11-15T05:56:37.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 msgr2=0x7f09141066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:37.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0908013510 tx=0x7f090800b040).stop 2023-11-15T05:56:37.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 shutdown_connections 2023-11-15T05:56:37.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f090402ff60 0x7f0904032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:37.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 --2- 172.21.15.137:0/4196517874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09140f6480 0x7f09141066d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:37.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 >> 172.21.15.137:0/4196517874 conn(0x7f09140f1c00 msgr2=0x7f09140f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:37.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 shutdown_connections 2023-11-15T05:56:37.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:37.727+0000 7f091b05d700 1 -- 172.21.15.137:0/4196517874 wait complete. 2023-11-15T05:56:37.731 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:38.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:36.981675+0000 mgr.smithi003.cldzke (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:37.726996+0000 mon.smithi003 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.137:0/4196517874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:39.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:39.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:40.311 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 -- 172.21.15.137:0/2891522916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 msgr2=0x7f9b300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:40.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 --2- 172.21.15.137:0/2891522916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b300f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9b20009fb0 tx=0x7f9b20009b30).stop 2023-11-15T05:56:40.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 -- 172.21.15.137:0/2891522916 shutdown_connections 2023-11-15T05:56:40.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 --2- 172.21.15.137:0/2891522916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b300f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:40.312 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 -- 172.21.15.137:0/2891522916 >> 172.21.15.137:0/2891522916 conn(0x7f9b300f0b70 msgr2=0x7f9b300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 -- 172.21.15.137:0/2891522916 shutdown_connections 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.308+0000 7f9b37d18700 1 -- 172.21.15.137:0/2891522916 wait complete. 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b37d18700 1 Processor -- start 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b37d18700 1 -- start start 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b37d18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:40.313 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b37d18700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30105af0 con 0x7f9b300f5410 2023-11-15T05:56:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b35ab4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b35ab4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:40784/0 (socket says 172.21.15.137:40784) 2023-11-15T05:56:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b35ab4700 1 -- 172.21.15.137:0/1097875534 learned_addr learned my addr 172.21.15.137:0/1097875534 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.309+0000 7f9b35ab4700 1 -- 172.21.15.137:0/1097875534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b20005040 con 0x7f9b300f5410 2023-11-15T05:56:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b35ab4700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9b20013510 tx=0x7f9b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:40.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b20009320 con 0x7f9b300f5410 2023-11-15T05:56:40.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b20013f00 con 0x7f9b300f5410 2023-11-15T05:56:40.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30105cf0 con 0x7f9b300f5410 2023-11-15T05:56:40.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b20014d90 con 0x7f9b300f5410 2023-11-15T05:56:40.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.310+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b30106180 con 0x7f9b300f5410 2023-11-15T05:56:40.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b20014420 con 0x7f9b300f5410 2023-11-15T05:56:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b30045ad0 con 0x7f9b300f5410 2023-11-15T05:56:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b267fc700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 0x7f9b1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b20040fb0 con 0x7f9b300f5410 2023-11-15T05:56:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b352b3700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 0x7f9b1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:40.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.311+0000 7f9b352b3700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 0x7f9b1c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9b2c0096a0 tx=0x7f9b2c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:40.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.315+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b200095f0 con 0x7f9b300f5410 2023-11-15T05:56:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:38.982000+0000 mgr.smithi003.cldzke (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.523+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b300fae60 con 0x7f9b300f5410 2023-11-15T05:56:40.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.524+0000 7f9b267fc700 1 -- 172.21.15.137:0/1097875534 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9b2001db70 con 0x7f9b300f5410 2023-11-15T05:56:40.526 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:40.526 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:40.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 msgr2=0x7f9b1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:40.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 0x7f9b1c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9b2c0096a0 tx=0x7f9b2c006b40).stop 2023-11-15T05:56:40.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 msgr2=0x7f9b30105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:40.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9b20013510 tx=0x7f9b2000b040).stop 2023-11-15T05:56:40.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 shutdown_connections 2023-11-15T05:56:40.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b1c02ff60 0x7f9b1c032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:40.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 --2- 172.21.15.137:0/1097875534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b300f5410 0x7f9b30105600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:40.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 >> 172.21.15.137:0/1097875534 conn(0x7f9b300f0b70 msgr2=0x7f9b300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:40.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 shutdown_connections 2023-11-15T05:56:40.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:40.526+0000 7f9b37d18700 1 -- 172.21.15.137:0/1097875534 wait complete. 2023-11-15T05:56:40.530 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:41.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:41.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:40.525632+0000 mon.smithi003 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.137:0/1097875534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:40.982338+0000 mgr.smithi003.cldzke (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:43.155 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- 172.21.15.137:0/3417137765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 msgr2=0x7ffb900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:43.155 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 --2- 172.21.15.137:0/3417137765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb900f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ffb80009fb0 tx=0x7ffb80009b30).stop 2023-11-15T05:56:43.155 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- 172.21.15.137:0/3417137765 shutdown_connections 2023-11-15T05:56:43.155 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 --2- 172.21.15.137:0/3417137765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb900f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:43.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- 172.21.15.137:0/3417137765 >> 172.21.15.137:0/3417137765 conn(0x7ffb900f0b70 msgr2=0x7ffb900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:43.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- 172.21.15.137:0/3417137765 shutdown_connections 2023-11-15T05:56:43.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- 172.21.15.137:0/3417137765 wait complete. 2023-11-15T05:56:43.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 Processor -- start 2023-11-15T05:56:43.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.152+0000 7ffb954a1700 1 -- start start 2023-11-15T05:56:43.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb954a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:43.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb954a1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb90107210 con 0x7ffb900f5410 2023-11-15T05:56:43.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:43.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:40802/0 (socket says 172.21.15.137:40802) 2023-11-15T05:56:43.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb8effd700 1 -- 172.21.15.137:0/2998810788 learned_addr learned my addr 172.21.15.137:0/2998810788 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:43.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb8effd700 1 -- 172.21.15.137:0/2998810788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80005040 con 0x7ffb900f5410 2023-11-15T05:56:43.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb8effd700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ffb80014000 tx=0x7ffb8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:43.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb800095f0 con 0x7ffb900f5410 2023-11-15T05:56:43.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb90060220 con 0x7ffb900f5410 2023-11-15T05:56:43.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb80007e70 con 0x7ffb900f5410 2023-11-15T05:56:43.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.153+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb80014ea0 con 0x7ffb900f5410 2023-11-15T05:56:43.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.154+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9010bd10 con 0x7ffb900f5410 2023-11-15T05:56:43.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.154+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb80025070 con 0x7ffb900f5410 2023-11-15T05:56:43.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.154+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb90045ad0 con 0x7ffb900f5410 2023-11-15T05:56:43.161 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.154+0000 7ffb877fe700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 0x7ffb7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:43.161 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.155+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb800409e0 con 0x7ffb900f5410 2023-11-15T05:56:43.161 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.155+0000 7ffb8e7fc700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 0x7ffb7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:43.161 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.155+0000 7ffb8e7fc700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 0x7ffb7c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ffb780096a0 tx=0x7ffb78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:43.162 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.159+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb800079c0 con 0x7ffb900f5410 2023-11-15T05:56:43.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.362+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb900fae60 con 0x7ffb900f5410 2023-11-15T05:56:43.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.362+0000 7ffb877fe700 1 -- 172.21.15.137:0/2998810788 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ffb8000f300 con 0x7ffb900f5410 2023-11-15T05:56:43.364 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:43.364 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:43.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 msgr2=0x7ffb7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:43.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 0x7ffb7c032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ffb780096a0 tx=0x7ffb78006b40).stop 2023-11-15T05:56:43.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 msgr2=0x7ffb9010b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:43.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ffb80014000 tx=0x7ffb8000b040).stop 2023-11-15T05:56:43.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 shutdown_connections 2023-11-15T05:56:43.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ffb7c02ff60 0x7ffb7c032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:43.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 --2- 172.21.15.137:0/2998810788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb900f5410 0x7ffb9010b820 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:43.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 >> 172.21.15.137:0/2998810788 conn(0x7ffb900f0b70 msgr2=0x7ffb900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:43.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 shutdown_connections 2023-11-15T05:56:43.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:43.364+0000 7ffb954a1700 1 -- 172.21.15.137:0/2998810788 wait complete. 2023-11-15T05:56:43.368 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:42.982564+0000 mgr.smithi003.cldzke (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:43.363811+0000 mon.smithi003 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.137:0/2998810788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:44.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:44.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:45.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 -- 172.21.15.137:0/4019744676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 msgr2=0x7ff99c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:45.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 --2- 172.21.15.137:0/4019744676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c0f55c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7ff98c009fb0 tx=0x7ff98c009b30).stop 2023-11-15T05:56:45.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 -- 172.21.15.137:0/4019744676 shutdown_connections 2023-11-15T05:56:45.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 --2- 172.21.15.137:0/4019744676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c0f55c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:45.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 -- 172.21.15.137:0/4019744676 >> 172.21.15.137:0/4019744676 conn(0x7ff99c0f09a0 msgr2=0x7ff99c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:45.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 -- 172.21.15.137:0/4019744676 shutdown_connections 2023-11-15T05:56:45.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 -- 172.21.15.137:0/4019744676 wait complete. 2023-11-15T05:56:45.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.992+0000 7ff99b59e700 1 Processor -- start 2023-11-15T05:56:45.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99b59e700 1 -- start start 2023-11-15T05:56:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1069d0 con 0x7ff99c0f51a0 2023-11-15T05:56:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:40832/0 (socket says 172.21.15.137:40832) 2023-11-15T05:56:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99a59c700 1 -- 172.21.15.137:0/1587185645 learned_addr learned my addr 172.21.15.137:0/1587185645 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:45.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99a59c700 1 -- 172.21.15.137:0/1587185645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c005040 con 0x7ff99c0f51a0 2023-11-15T05:56:45.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff99a59c700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff98c013510 tx=0x7ff98c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:45.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.993+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c009320 con 0x7ff99c0f51a0 2023-11-15T05:56:45.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c106bd0 con 0x7ff99c0f51a0 2023-11-15T05:56:45.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff98c013f00 con 0x7ff99c0f51a0 2023-11-15T05:56:46.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c014dc0 con 0x7ff99c0f51a0 2023-11-15T05:56:46.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c107060 con 0x7ff99c0f51a0 2023-11-15T05:56:46.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff98c0095f0 con 0x7ff99c0f51a0 2023-11-15T05:56:46.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c045ad0 con 0x7ff99c0f51a0 2023-11-15T05:56:46.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff98affd700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 0x7ff984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:46.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.994+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff98c041020 con 0x7ff99c0f51a0 2023-11-15T05:56:46.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.995+0000 7ff999d9b700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 0x7ff984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:46.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.995+0000 7ff999d9b700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 0x7ff984032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff9900096a0 tx=0x7ff990006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:46.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:45.999+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff98c014420 con 0x7ff99c0f51a0 2023-11-15T05:56:46.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.202+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff99c107f10 con 0x7ff99c0f51a0 2023-11-15T05:56:46.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.203+0000 7ff98affd700 1 -- 172.21.15.137:0/1587185645 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff98c01c920 con 0x7ff99c0f51a0 2023-11-15T05:56:46.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:46.205 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:46.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.204+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 msgr2=0x7ff984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:46.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.204+0000 7ff99b59e700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 0x7ff984032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff9900096a0 tx=0x7ff990006b40).stop 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.204+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 msgr2=0x7ff99c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.204+0000 7ff99b59e700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7ff98c013510 tx=0x7ff98c00b040).stop 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 shutdown_connections 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff98402ff60 0x7ff984032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 --2- 172.21.15.137:0/1587185645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff99c0f51a0 0x7ff99c1064e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 >> 172.21.15.137:0/1587185645 conn(0x7ff99c0f09a0 msgr2=0x7ff99c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:46.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 shutdown_connections 2023-11-15T05:56:46.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:46.205+0000 7ff99b59e700 1 -- 172.21.15.137:0/1587185645 wait complete. 2023-11-15T05:56:46.208 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:44.982900+0000 mgr.smithi003.cldzke (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:47.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:46.204437+0000 mon.smithi003 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.137:0/1587185645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:47.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:47.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:48.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:46.983218+0000 mgr.smithi003.cldzke (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:48.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.757+0000 7fecb8e45700 1 -- 172.21.15.137:0/1262656326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 msgr2=0x7fecb40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:48.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.757+0000 7fecb8e45700 1 --2- 172.21.15.137:0/1262656326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb40f55c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7feca4009fb0 tx=0x7feca4009b30).stop 2023-11-15T05:56:48.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.757+0000 7fecb8e45700 1 -- 172.21.15.137:0/1262656326 shutdown_connections 2023-11-15T05:56:48.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.757+0000 7fecb8e45700 1 --2- 172.21.15.137:0/1262656326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb40f55c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:48.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.757+0000 7fecb8e45700 1 -- 172.21.15.137:0/1262656326 >> 172.21.15.137:0/1262656326 conn(0x7fecb40f09a0 msgr2=0x7fecb40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:48.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 -- 172.21.15.137:0/1262656326 shutdown_connections 2023-11-15T05:56:48.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 -- 172.21.15.137:0/1262656326 wait complete. 2023-11-15T05:56:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 Processor -- start 2023-11-15T05:56:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 -- start start 2023-11-15T05:56:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.758+0000 7fecb8e45700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb41058a0 con 0x7fecb40f51a0 2023-11-15T05:56:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:48.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:40844/0 (socket says 172.21.15.137:40844) 2023-11-15T05:56:48.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb37fe700 1 -- 172.21.15.137:0/3259056128 learned_addr learned my addr 172.21.15.137:0/3259056128 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:48.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb37fe700 1 -- 172.21.15.137:0/3259056128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4005040 con 0x7fecb40f51a0 2023-11-15T05:56:48.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb37fe700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7feca4013510 tx=0x7feca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:48.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feca4009320 con 0x7fecb40f51a0 2023-11-15T05:56:48.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb4105aa0 con 0x7fecb40f51a0 2023-11-15T05:56:48.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feca4013f00 con 0x7fecb40f51a0 2023-11-15T05:56:48.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feca4014dc0 con 0x7fecb40f51a0 2023-11-15T05:56:48.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.759+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb4105f30 con 0x7fecb40f51a0 2023-11-15T05:56:48.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.760+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feca40095f0 con 0x7fecb40f51a0 2023-11-15T05:56:48.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.760+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4045ad0 con 0x7fecb40f51a0 2023-11-15T05:56:48.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.761+0000 7fec9bfff700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 0x7fec9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:48.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.761+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feca4041020 con 0x7fecb40f51a0 2023-11-15T05:56:48.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.761+0000 7fecb2ffd700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 0x7fec9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:48.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.761+0000 7fecb2ffd700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 0x7fec9c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feca80096a0 tx=0x7feca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:48.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.765+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feca4014420 con 0x7fecb40f51a0 2023-11-15T05:56:48.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.973+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecb4106d80 con 0x7fecb40f51a0 2023-11-15T05:56:48.975 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.973+0000 7fec9bfff700 1 -- 172.21.15.137:0/3259056128 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7feca401c920 con 0x7fecb40f51a0 2023-11-15T05:56:48.975 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:48.976 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:48.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 msgr2=0x7fec9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:48.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 0x7fec9c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7feca80096a0 tx=0x7feca8006b40).stop 2023-11-15T05:56:48.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 msgr2=0x7fecb41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:48.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7feca4013510 tx=0x7feca400b040).stop 2023-11-15T05:56:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 shutdown_connections 2023-11-15T05:56:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fec9c02ff60 0x7fec9c032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 --2- 172.21.15.137:0/3259056128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecb40f51a0 0x7fecb41053b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 >> 172.21.15.137:0/3259056128 conn(0x7fecb40f09a0 msgr2=0x7fecb40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 shutdown_connections 2023-11-15T05:56:48.979 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:48.975+0000 7fecb8e45700 1 -- 172.21.15.137:0/3259056128 wait complete. 2023-11-15T05:56:48.979 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:49.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:49 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:48.974835+0000 mon.smithi003 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.137:0/3259056128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:50.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:50.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:50.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:48.983489+0000 mgr.smithi003.cldzke (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:51.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 -- 172.21.15.137:0/564387210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 msgr2=0x7f3f500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:51.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 --2- 172.21.15.137:0/564387210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f500f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3f40009fb0 tx=0x7f3f40009b30).stop 2023-11-15T05:56:51.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 -- 172.21.15.137:0/564387210 shutdown_connections 2023-11-15T05:56:51.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 --2- 172.21.15.137:0/564387210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f500f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:51.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 -- 172.21.15.137:0/564387210 >> 172.21.15.137:0/564387210 conn(0x7f3f500f0b70 msgr2=0x7f3f500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:51.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 -- 172.21.15.137:0/564387210 shutdown_connections 2023-11-15T05:56:51.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 -- 172.21.15.137:0/564387210 wait complete. 2023-11-15T05:56:51.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.579+0000 7f3f562be700 1 Processor -- start 2023-11-15T05:56:51.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f562be700 1 -- start start 2023-11-15T05:56:51.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f562be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:51.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f562be700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50060220 con 0x7f3f500f5410 2023-11-15T05:56:51.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50334/0 (socket says 172.21.15.137:50334) 2023-11-15T05:56:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f4ffff700 1 -- 172.21.15.137:0/473237596 learned_addr learned my addr 172.21.15.137:0/473237596 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f4ffff700 1 -- 172.21.15.137:0/473237596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40005040 con 0x7f3f500f5410 2023-11-15T05:56:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.580+0000 7f3f4ffff700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3f40013510 tx=0x7f3f4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f40009320 con 0x7f3f500f5410 2023-11-15T05:56:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f40013f00 con 0x7f3f500f5410 2023-11-15T05:56:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f50060420 con 0x7f3f500f5410 2023-11-15T05:56:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f40014d90 con 0x7f3f500f5410 2023-11-15T05:56:51.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f50109b00 con 0x7f3f500f5410 2023-11-15T05:56:51.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.581+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f400095f0 con 0x7f3f500f5410 2023-11-15T05:56:51.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.582+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f50059100 con 0x7f3f500f5410 2023-11-15T05:56:51.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.582+0000 7f3f4cff9700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 0x7f3f38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:51.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.582+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f400410c0 con 0x7f3f500f5410 2023-11-15T05:56:51.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.582+0000 7f3f4f7fe700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 0x7f3f38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:51.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.582+0000 7f3f4f7fe700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 0x7f3f38032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:51.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.587+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f40014420 con 0x7f3f500f5410 2023-11-15T05:56:51.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.793+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f50045ad0 con 0x7f3f500f5410 2023-11-15T05:56:51.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.793+0000 7f3f4cff9700 1 -- 172.21.15.137:0/473237596 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3f4001d7f0 con 0x7f3f500f5410 2023-11-15T05:56:51.796 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:51.796 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:51.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 msgr2=0x7f3f38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:51.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 0x7f3f38032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).stop 2023-11-15T05:56:51.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 msgr2=0x7f3f50109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:51.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3f40013510 tx=0x7f3f4000b040).stop 2023-11-15T05:56:51.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 shutdown_connections 2023-11-15T05:56:51.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f3802ff60 0x7f3f38032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:51.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 --2- 172.21.15.137:0/473237596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f500f5410 0x7f3f50109450 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:51.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 >> 172.21.15.137:0/473237596 conn(0x7f3f500f0b70 msgr2=0x7f3f500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:51.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 shutdown_connections 2023-11-15T05:56:51.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:51.795+0000 7f3f562be700 1 -- 172.21.15.137:0/473237596 wait complete. 2023-11-15T05:56:51.801 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:51.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:50.983869+0000 mgr.smithi003.cldzke (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:52.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:51.794728+0000 mon.smithi003 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.137:0/473237596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:53.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:53.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:53.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:52.984216+0000 mgr.smithi003.cldzke (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:54.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 -- 172.21.15.137:0/2323498535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 msgr2=0x7feff40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:54.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 --2- 172.21.15.137:0/2323498535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff40f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fefe8009fb0 tx=0x7fefe8009b30).stop 2023-11-15T05:56:54.391 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 -- 172.21.15.137:0/2323498535 shutdown_connections 2023-11-15T05:56:54.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 --2- 172.21.15.137:0/2323498535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff40f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:54.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 -- 172.21.15.137:0/2323498535 >> 172.21.15.137:0/2323498535 conn(0x7feff40f09a0 msgr2=0x7feff40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:54.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 -- 172.21.15.137:0/2323498535 shutdown_connections 2023-11-15T05:56:54.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.388+0000 7feff996b700 1 -- 172.21.15.137:0/2323498535 wait complete. 2023-11-15T05:56:54.392 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff996b700 1 Processor -- start 2023-11-15T05:56:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff996b700 1 -- start start 2023-11-15T05:56:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff996b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff996b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff41058a0 con 0x7feff40f51a0 2023-11-15T05:56:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff8969700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff8969700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50354/0 (socket says 172.21.15.137:50354) 2023-11-15T05:56:54.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.389+0000 7feff8969700 1 -- 172.21.15.137:0/3779370498 learned_addr learned my addr 172.21.15.137:0/3779370498 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:54.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff8969700 1 -- 172.21.15.137:0/3779370498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8005040 con 0x7feff40f51a0 2023-11-15T05:56:54.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff8969700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fefe8013510 tx=0x7fefe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:54.396 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe8009320 con 0x7feff40f51a0 2023-11-15T05:56:54.396 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4105aa0 con 0x7feff40f51a0 2023-11-15T05:56:54.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefe8013f00 con 0x7feff40f51a0 2023-11-15T05:56:54.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefe8014dc0 con 0x7feff40f51a0 2023-11-15T05:56:54.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.390+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff4105f30 con 0x7feff40f51a0 2023-11-15T05:56:54.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.391+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefe8014420 con 0x7feff40f51a0 2023-11-15T05:56:54.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.391+0000 7feff17fa700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:54.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.391+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4045ad0 con 0x7feff40f51a0 2023-11-15T05:56:54.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.391+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefe8041030 con 0x7feff40f51a0 2023-11-15T05:56:54.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.391+0000 7feff3fff700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:54.399 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.392+0000 7feff3fff700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 0x7fefe4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:54.399 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.396+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefe80095f0 con 0x7feff40f51a0 2023-11-15T05:56:54.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.601+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feff4106de0 con 0x7feff40f51a0 2023-11-15T05:56:54.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.601+0000 7feff17fa700 1 -- 172.21.15.137:0/3779370498 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fefe802c090 con 0x7feff40f51a0 2023-11-15T05:56:54.603 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:54.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 msgr2=0x7fefe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:54.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 0x7fefe4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fefe00096a0 tx=0x7fefe0006b40).stop 2023-11-15T05:56:54.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 msgr2=0x7feff41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:54.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fefe8013510 tx=0x7fefe800b040).stop 2023-11-15T05:56:54.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 shutdown_connections 2023-11-15T05:56:54.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:54.605 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 --2- 172.21.15.137:0/3779370498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feff40f51a0 0x7feff41053b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:54.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 >> 172.21.15.137:0/3779370498 conn(0x7feff40f09a0 msgr2=0x7feff40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:54.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 shutdown_connections 2023-11-15T05:56:54.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:54.603+0000 7feff996b700 1 -- 172.21.15.137:0/3779370498 wait complete. 2023-11-15T05:56:54.607 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:54.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:54.602633+0000 mon.smithi003 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.137:0/3779370498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:55.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:55 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:54.984534+0000 mgr.smithi003.cldzke (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:56.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:56.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:57.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- 172.21.15.137:0/4275850401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 msgr2=0x7f5e8c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:57.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 --2- 172.21.15.137:0/4275850401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c0f55c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5e7c009fb0 tx=0x7f5e7c009b30).stop 2023-11-15T05:56:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- 172.21.15.137:0/4275850401 shutdown_connections 2023-11-15T05:56:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 --2- 172.21.15.137:0/4275850401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c0f55c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- 172.21.15.137:0/4275850401 >> 172.21.15.137:0/4275850401 conn(0x7f5e8c0f09a0 msgr2=0x7f5e8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- 172.21.15.137:0/4275850401 shutdown_connections 2023-11-15T05:56:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- 172.21.15.137:0/4275850401 wait complete. 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 Processor -- start 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- start start 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.144+0000 7f5e916cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c1069b0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:57.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50372/0 (socket says 172.21.15.137:50372) 2023-11-15T05:56:57.149 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e8bfff700 1 -- 172.21.15.137:0/1799812837 learned_addr learned my addr 172.21.15.137:0/1799812837 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:57.150 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e8bfff700 1 -- 172.21.15.137:0/1799812837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c005040 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.150 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e8bfff700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c009320 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c106bb0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e7c013f00 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7c014dc0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.145+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c107040 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.146+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e7c0263a0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.146+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c045ad0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.146+0000 7f5e88ff9700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 0x7f5e74032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:57.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.146+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e7c040fe0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.146+0000 7f5e8b7fe700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 0x7f5e74032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:57.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.147+0000 7f5e8b7fe700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 0x7f5e74032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5e800096a0 tx=0x7f5e80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:57.154 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.151+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e7c06d2b0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.360 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.358+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e8c107ef0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.359+0000 7f5e88ff9700 1 -- 172.21.15.137:0/1799812837 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5e7c0126f0 con 0x7f5e8c0f51a0 2023-11-15T05:56:57.361 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:56:57.361 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:56:57.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 msgr2=0x7f5e74032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:57.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 0x7f5e74032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5e800096a0 tx=0x7f5e80006b40).stop 2023-11-15T05:56:57.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 msgr2=0x7f5e8c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:57.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5e7c013510 tx=0x7f5e7c00b040).stop 2023-11-15T05:56:57.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 shutdown_connections 2023-11-15T05:56:57.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5e7402ffb0 0x7f5e74032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:57.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 --2- 172.21.15.137:0/1799812837 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5e8c0f51a0 0x7f5e8c1064c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:57.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 >> 172.21.15.137:0/1799812837 conn(0x7f5e8c0f09a0 msgr2=0x7f5e8c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:57.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 shutdown_connections 2023-11-15T05:56:57.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:57.361+0000 7f5e916cf700 1 -- 172.21.15.137:0/1799812837 wait complete. 2023-11-15T05:56:57.364 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:56:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:56.984781+0000 mgr.smithi003.cldzke (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:56:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:56:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:56:57.360484+0000 mon.smithi003 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.137:0/1799812837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:56:58.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:56:58.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:56:59.929 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 -- 172.21.15.137:0/764287574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 msgr2=0x7fe2d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:56:59.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/764287574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe2c4009fb0 tx=0x7fe2c4009b30).stop 2023-11-15T05:56:59.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 -- 172.21.15.137:0/764287574 shutdown_connections 2023-11-15T05:56:59.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/764287574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:56:59.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 -- 172.21.15.137:0/764287574 >> 172.21.15.137:0/764287574 conn(0x7fe2d40f0b70 msgr2=0x7fe2d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:56:59.930 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 -- 172.21.15.137:0/764287574 shutdown_connections 2023-11-15T05:56:59.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 -- 172.21.15.137:0/764287574 wait complete. 2023-11-15T05:56:59.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.927+0000 7fe2d9bec700 1 Processor -- start 2023-11-15T05:56:59.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d9bec700 1 -- start start 2023-11-15T05:56:59.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d9bec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:59.931 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d9bec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d40fb820 con 0x7fe2d40f5410 2023-11-15T05:56:59.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:59.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56166/0 (socket says 172.21.15.137:56166) 2023-11-15T05:56:59.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d37fe700 1 -- 172.21.15.137:0/3964952699 learned_addr learned my addr 172.21.15.137:0/3964952699 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:56:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d37fe700 1 -- 172.21.15.137:0/3964952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4005040 con 0x7fe2d40f5410 2023-11-15T05:56:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.928+0000 7fe2d37fe700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe2c4014000 tx=0x7fe2c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2c40095f0 con 0x7fe2d40f5410 2023-11-15T05:56:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d40fb9c0 con 0x7fe2d40f5410 2023-11-15T05:56:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2c4007e70 con 0x7fe2d40f5410 2023-11-15T05:56:59.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2c4014ea0 con 0x7fe2d40f5410 2023-11-15T05:56:59.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d40fbe50 con 0x7fe2d40f5410 2023-11-15T05:56:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2c40079c0 con 0x7fe2d40f5410 2023-11-15T05:56:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.929+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d4059100 con 0x7fe2d40f5410 2023-11-15T05:56:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.930+0000 7fe2cbfff700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 0x7fe2c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:56:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.930+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2c40408e0 con 0x7fe2d40f5410 2023-11-15T05:56:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.930+0000 7fe2d2ffd700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 0x7fe2c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:56:59.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.930+0000 7fe2d2ffd700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 0x7fe2c0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2bc0096a0 tx=0x7fe2bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:56:59.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:56:59.935+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2c400f950 con 0x7fe2d40f5410 2023-11-15T05:57:00.142 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.140+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2d40fae60 con 0x7fe2d40f5410 2023-11-15T05:57:00.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.141+0000 7fe2cbfff700 1 -- 172.21.15.137:0/3964952699 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe2c4010230 con 0x7fe2d40f5410 2023-11-15T05:57:00.143 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:00.143 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 msgr2=0x7fe2c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 0x7fe2c0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe2bc0096a0 tx=0x7fe2bc006b40).stop 2023-11-15T05:57:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 msgr2=0x7fe2d40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:00.145 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe2c4014000 tx=0x7fe2c400b040).stop 2023-11-15T05:57:00.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 shutdown_connections 2023-11-15T05:57:00.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2c002ff60 0x7fe2c0032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:00.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 --2- 172.21.15.137:0/3964952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2d40f5410 0x7fe2d40fd3a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:00.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 >> 172.21.15.137:0/3964952699 conn(0x7fe2d40f0b70 msgr2=0x7fe2d405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:00.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 shutdown_connections 2023-11-15T05:57:00.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:00.143+0000 7fe2d9bec700 1 -- 172.21.15.137:0/3964952699 wait complete. 2023-11-15T05:57:00.147 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:56:58.985113+0000 mgr.smithi003.cldzke (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:01.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:00.142636+0000 mon.smithi003 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.137:0/3964952699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:01.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:01.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:02.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:00.985331+0000 mgr.smithi003.cldzke (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:02.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 -- 172.21.15.137:0/2075953238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 msgr2=0x7f77440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:02.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 --2- 172.21.15.137:0/2075953238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f77440f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f7738009fb0 tx=0x7f7738009b30).stop 2023-11-15T05:57:02.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 -- 172.21.15.137:0/2075953238 shutdown_connections 2023-11-15T05:57:02.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 --2- 172.21.15.137:0/2075953238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f77440f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:02.722 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 -- 172.21.15.137:0/2075953238 >> 172.21.15.137:0/2075953238 conn(0x7f77440f0b70 msgr2=0x7f77440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:02.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 -- 172.21.15.137:0/2075953238 shutdown_connections 2023-11-15T05:57:02.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.719+0000 7f774aa7b700 1 -- 172.21.15.137:0/2075953238 wait complete. 2023-11-15T05:57:02.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f774aa7b700 1 Processor -- start 2023-11-15T05:57:02.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f774aa7b700 1 -- start start 2023-11-15T05:57:02.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f774aa7b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:02.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f774aa7b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7744105b20 con 0x7f77440f5410 2023-11-15T05:57:02.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f7748817700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f7748817700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56194/0 (socket says 172.21.15.137:56194) 2023-11-15T05:57:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.720+0000 7f7748817700 1 -- 172.21.15.137:0/4185776243 learned_addr learned my addr 172.21.15.137:0/4185776243 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f7748817700 1 -- 172.21.15.137:0/4185776243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738005040 con 0x7f77440f5410 2023-11-15T05:57:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f7748817700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7738014000 tx=0x7f773800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77380095f0 con 0x7f77440f5410 2023-11-15T05:57:02.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7738007e70 con 0x7f77440f5410 2023-11-15T05:57:02.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7744105d20 con 0x7f77440f5410 2023-11-15T05:57:02.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7738014ea0 con 0x7f77440f5410 2023-11-15T05:57:02.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.721+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77441061b0 con 0x7f77440f5410 2023-11-15T05:57:02.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.722+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7738025070 con 0x7f77440f5410 2023-11-15T05:57:02.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.722+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7744045ad0 con 0x7f77440f5410 2023-11-15T05:57:02.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.722+0000 7f77417fa700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:02.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.722+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77380409f0 con 0x7f77440f5410 2023-11-15T05:57:02.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.723+0000 7f7743fff700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:02.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.723+0000 7f7743fff700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:02.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.728+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77380079c0 con 0x7f77440f5410 2023-11-15T05:57:02.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.932+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77440fae60 con 0x7f77440f5410 2023-11-15T05:57:02.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.932+0000 7f77417fa700 1 -- 172.21.15.137:0/4185776243 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f773800f300 con 0x7f77440f5410 2023-11-15T05:57:02.934 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:02.935 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:02.935 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 msgr2=0x7f7734032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:02.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).stop 2023-11-15T05:57:02.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 msgr2=0x7f7744105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:02.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7738014000 tx=0x7f773800b040).stop 2023-11-15T05:57:02.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 shutdown_connections 2023-11-15T05:57:02.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:02.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 --2- 172.21.15.137:0/4185776243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77440f5410 0x7f7744105630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:02.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 >> 172.21.15.137:0/4185776243 conn(0x7f77440f0b70 msgr2=0x7f77440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:02.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 shutdown_connections 2023-11-15T05:57:02.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:02.934+0000 7f774aa7b700 1 -- 172.21.15.137:0/4185776243 wait complete. 2023-11-15T05:57:02.938 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:03.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:02.933741+0000 mon.smithi003 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.137:0/4185776243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:04.344 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:04.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:04.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:02.985676+0000 mgr.smithi003.cldzke (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:05.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- 172.21.15.137:0/3871981742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 msgr2=0x7f94980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:05.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 --2- 172.21.15.137:0/3871981742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94980f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f948c009fb0 tx=0x7f948c009b30).stop 2023-11-15T05:57:05.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- 172.21.15.137:0/3871981742 shutdown_connections 2023-11-15T05:57:05.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 --2- 172.21.15.137:0/3871981742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94980f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:05.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- 172.21.15.137:0/3871981742 >> 172.21.15.137:0/3871981742 conn(0x7f94980f09a0 msgr2=0x7f94980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:05.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- 172.21.15.137:0/3871981742 shutdown_connections 2023-11-15T05:57:05.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- 172.21.15.137:0/3871981742 wait complete. 2023-11-15T05:57:05.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 Processor -- start 2023-11-15T05:57:05.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.534+0000 7f949d9d0700 1 -- start start 2023-11-15T05:57:05.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949d9d0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:05.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949d9d0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9498060240 con 0x7f94980f51a0 2023-11-15T05:57:05.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949c9ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:05.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949c9ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56206/0 (socket says 172.21.15.137:56206) 2023-11-15T05:57:05.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949c9ce700 1 -- 172.21.15.137:0/827935435 learned_addr learned my addr 172.21.15.137:0/827935435 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:05.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949c9ce700 1 -- 172.21.15.137:0/827935435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c005040 con 0x7f94980f51a0 2023-11-15T05:57:05.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.535+0000 7f949c9ce700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f948c013510 tx=0x7f948c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:05.542 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c009320 con 0x7f94980f51a0 2023-11-15T05:57:05.542 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9498060440 con 0x7f94980f51a0 2023-11-15T05:57:05.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f948c013f00 con 0x7f94980f51a0 2023-11-15T05:57:05.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f948c014dc0 con 0x7f94980f51a0 2023-11-15T05:57:05.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9498109860 con 0x7f94980f51a0 2023-11-15T05:57:05.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.536+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f948c014420 con 0x7f94980f51a0 2023-11-15T05:57:05.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.537+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9498045ad0 con 0x7f94980f51a0 2023-11-15T05:57:05.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.537+0000 7f94957fa700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 0x7f9488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:05.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.537+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f948c041030 con 0x7f94980f51a0 2023-11-15T05:57:05.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.537+0000 7f9497fff700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 0x7f9488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:05.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.537+0000 7f9497fff700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 0x7f9488032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f94840096a0 tx=0x7f9484006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:05.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.541+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f948c0095f0 con 0x7f94980f51a0 2023-11-15T05:57:05.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.746+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f949810a080 con 0x7f94980f51a0 2023-11-15T05:57:05.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.747+0000 7f94957fa700 1 -- 172.21.15.137:0/827935435 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f948c01cb10 con 0x7f94980f51a0 2023-11-15T05:57:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:05.750 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:05.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 msgr2=0x7f9488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:05.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 0x7f9488032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f94840096a0 tx=0x7f9484006b40).stop 2023-11-15T05:57:05.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 msgr2=0x7f94981091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:05.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f948c013510 tx=0x7f948c00b040).stop 2023-11-15T05:57:05.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 shutdown_connections 2023-11-15T05:57:05.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f948802ff60 0x7f9488032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:05.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 --2- 172.21.15.137:0/827935435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94980f51a0 0x7f94981091b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:05.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 >> 172.21.15.137:0/827935435 conn(0x7f94980f09a0 msgr2=0x7f94980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:05.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 shutdown_connections 2023-11-15T05:57:05.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:05.748+0000 7f949d9d0700 1 -- 172.21.15.137:0/827935435 wait complete. 2023-11-15T05:57:05.754 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:04.985969+0000 mgr.smithi003.cldzke (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:05.748144+0000 mon.smithi003 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.137:0/827935435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:07.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:07.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:08.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 -- 172.21.15.137:0/4286496108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 msgr2=0x7f44a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:08.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 --2- 172.21.15.137:0/4286496108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a00f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f4490009fb0 tx=0x7f4490009b30).stop 2023-11-15T05:57:08.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 -- 172.21.15.137:0/4286496108 shutdown_connections 2023-11-15T05:57:08.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 --2- 172.21.15.137:0/4286496108 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a00f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:08.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 -- 172.21.15.137:0/4286496108 >> 172.21.15.137:0/4286496108 conn(0x7f44a00f0b70 msgr2=0x7f44a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 -- 172.21.15.137:0/4286496108 shutdown_connections 2023-11-15T05:57:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.350+0000 7f44a6351700 1 -- 172.21.15.137:0/4286496108 wait complete. 2023-11-15T05:57:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.351+0000 7f44a6351700 1 Processor -- start 2023-11-15T05:57:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.351+0000 7f44a6351700 1 -- start start 2023-11-15T05:57:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.351+0000 7f44a6351700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:08.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.351+0000 7f44a6351700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0060240 con 0x7f44a00f5410 2023-11-15T05:57:08.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:08.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56214/0 (socket says 172.21.15.137:56214) 2023-11-15T05:57:08.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449ffff700 1 -- 172.21.15.137:0/3138412931 learned_addr learned my addr 172.21.15.137:0/3138412931 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:08.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449ffff700 1 -- 172.21.15.137:0/3138412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490005040 con 0x7f44a00f5410 2023-11-15T05:57:08.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449ffff700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4490013510 tx=0x7f449000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:08.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490009320 con 0x7f44a00f5410 2023-11-15T05:57:08.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a0060440 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4490013f00 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490014d90 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.352+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a0109ae0 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.353+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4490014420 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.353+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a0045ad0 con 0x7f44a00f5410 2023-11-15T05:57:08.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.353+0000 7f449cff9700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 0x7f4488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:08.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.353+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44900410c0 con 0x7f44a00f5410 2023-11-15T05:57:08.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.353+0000 7f449f7fe700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 0x7f4488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:08.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.354+0000 7f449f7fe700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 0x7f4488032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f44940096a0 tx=0x7f4494006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:08.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.358+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44900095f0 con 0x7f44a00f5410 2023-11-15T05:57:08.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:06.986290+0000 mgr.smithi003.cldzke (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:08.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.561+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44a00fae60 con 0x7f44a00f5410 2023-11-15T05:57:08.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.562+0000 7f449cff9700 1 -- 172.21.15.137:0/3138412931 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f449001db70 con 0x7f44a00f5410 2023-11-15T05:57:08.564 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:08.564 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:08.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 msgr2=0x7f4488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:08.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 0x7f4488032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f44940096a0 tx=0x7f4494006b40).stop 2023-11-15T05:57:08.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 msgr2=0x7f44a0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:08.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4490013510 tx=0x7f449000b040).stop 2023-11-15T05:57:08.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 shutdown_connections 2023-11-15T05:57:08.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f448802ff60 0x7f4488032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:08.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 --2- 172.21.15.137:0/3138412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44a00f5410 0x7f44a0109430 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:08.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 >> 172.21.15.137:0/3138412931 conn(0x7f44a00f0b70 msgr2=0x7f44a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:08.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 shutdown_connections 2023-11-15T05:57:08.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:08.564+0000 7f44a6351700 1 -- 172.21.15.137:0/3138412931 wait complete. 2023-11-15T05:57:08.568 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:09.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:08.563572+0000 mon.smithi003 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.137:0/3138412931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:09.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:09.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:10.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:08.986676+0000 mgr.smithi003.cldzke (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:11.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- 172.21.15.137:0/1182348901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 msgr2=0x7f26d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:11.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 --2- 172.21.15.137:0/1182348901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d40f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f26c8009fb0 tx=0x7f26c8009b30).stop 2023-11-15T05:57:11.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- 172.21.15.137:0/1182348901 shutdown_connections 2023-11-15T05:57:11.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 --2- 172.21.15.137:0/1182348901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d40f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:11.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- 172.21.15.137:0/1182348901 >> 172.21.15.137:0/1182348901 conn(0x7f26d40f0b70 msgr2=0x7f26d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:11.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- 172.21.15.137:0/1182348901 shutdown_connections 2023-11-15T05:57:11.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- 172.21.15.137:0/1182348901 wait complete. 2023-11-15T05:57:11.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 Processor -- start 2023-11-15T05:57:11.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.108+0000 7f26db08a700 1 -- start start 2023-11-15T05:57:11.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26db08a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:11.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26db08a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4106c00 con 0x7f26d40f5410 2023-11-15T05:57:11.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26d8e26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:11.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26d8e26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56566/0 (socket says 172.21.15.137:56566) 2023-11-15T05:57:11.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26d8e26700 1 -- 172.21.15.137:0/1666098989 learned_addr learned my addr 172.21.15.137:0/1666098989 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:11.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26d8e26700 1 -- 172.21.15.137:0/1666098989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c8005040 con 0x7f26d40f5410 2023-11-15T05:57:11.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.109+0000 7f26d8e26700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f26c8014000 tx=0x7f26c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:11.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c80095f0 con 0x7f26d40f5410 2023-11-15T05:57:11.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d4106e00 con 0x7f26d40f5410 2023-11-15T05:57:11.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26c8007e70 con 0x7f26d40f5410 2023-11-15T05:57:11.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26c8014ea0 con 0x7f26d40f5410 2023-11-15T05:57:11.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d4107290 con 0x7f26d40f5410 2023-11-15T05:57:11.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26c8025070 con 0x7f26d40f5410 2023-11-15T05:57:11.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.110+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d4045ad0 con 0x7f26d40f5410 2023-11-15T05:57:11.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.111+0000 7f26d17fa700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 0x7f26c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:11.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.111+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26c80409f0 con 0x7f26d40f5410 2023-11-15T05:57:11.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.111+0000 7f26d3fff700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 0x7f26c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:11.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.111+0000 7f26d3fff700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 0x7f26c4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f26c00096a0 tx=0x7f26c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:11.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.115+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26c80079c0 con 0x7f26d40f5410 2023-11-15T05:57:11.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.322+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26d4108140 con 0x7f26d40f5410 2023-11-15T05:57:11.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.323+0000 7f26d17fa700 1 -- 172.21.15.137:0/1666098989 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f26c800f300 con 0x7f26d40f5410 2023-11-15T05:57:11.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:11.325 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:11.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 msgr2=0x7f26c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:11.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 0x7f26c4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f26c00096a0 tx=0x7f26c0006b40).stop 2023-11-15T05:57:11.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 msgr2=0x7f26d4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:11.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f26c8014000 tx=0x7f26c800b040).stop 2023-11-15T05:57:11.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 shutdown_connections 2023-11-15T05:57:11.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f26c402ff60 0x7f26c4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:11.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 --2- 172.21.15.137:0/1666098989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26d40f5410 0x7f26d4106710 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:11.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 >> 172.21.15.137:0/1666098989 conn(0x7f26d40f0b70 msgr2=0x7f26d405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:11.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.324+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 shutdown_connections 2023-11-15T05:57:11.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:11.325+0000 7f26db08a700 1 -- 172.21.15.137:0/1666098989 wait complete. 2023-11-15T05:57:11.328 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:10.986975+0000 mgr.smithi003.cldzke (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:11.324030+0000 mon.smithi003 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.137:0/1666098989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:12.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:12.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:13.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 -- 172.21.15.137:0/2659333199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 msgr2=0x7f5fd00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:13.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 --2- 172.21.15.137:0/2659333199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd00f4be0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5fc4009fb0 tx=0x7f5fc4009b30).stop 2023-11-15T05:57:13.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 -- 172.21.15.137:0/2659333199 shutdown_connections 2023-11-15T05:57:13.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 --2- 172.21.15.137:0/2659333199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd00f4be0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:13.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 -- 172.21.15.137:0/2659333199 >> 172.21.15.137:0/2659333199 conn(0x7f5fd00f02e0 msgr2=0x7f5fd00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:13.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 -- 172.21.15.137:0/2659333199 shutdown_connections 2023-11-15T05:57:13.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.916+0000 7f5fd5356700 1 -- 172.21.15.137:0/2659333199 wait complete. 2023-11-15T05:57:13.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.917+0000 7f5fd5356700 1 Processor -- start 2023-11-15T05:57:13.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.917+0000 7f5fd5356700 1 -- start start 2023-11-15T05:57:13.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.917+0000 7f5fd5356700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:13.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fd5356700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0106bf0 con 0x7f5fd00f47c0 2023-11-15T05:57:13.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fceffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:13.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fceffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56580/0 (socket says 172.21.15.137:56580) 2023-11-15T05:57:13.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fceffd700 1 -- 172.21.15.137:0/968792078 learned_addr learned my addr 172.21.15.137:0/968792078 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:13.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fceffd700 1 -- 172.21.15.137:0/968792078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc4005040 con 0x7f5fd00f47c0 2023-11-15T05:57:13.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fceffd700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5fc4013510 tx=0x7f5fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:13.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.918+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc4009320 con 0x7f5fd00f47c0 2023-11-15T05:57:13.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd0106df0 con 0x7f5fd00f47c0 2023-11-15T05:57:13.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fc4013f00 con 0x7f5fd00f47c0 2023-11-15T05:57:13.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc4014d90 con 0x7f5fd00f47c0 2023-11-15T05:57:13.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd0107280 con 0x7f5fd00f47c0 2023-11-15T05:57:13.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fc40095f0 con 0x7f5fd00f47c0 2023-11-15T05:57:13.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.919+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd0045ad0 con 0x7f5fd00f47c0 2023-11-15T05:57:13.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.920+0000 7f5fb77fe700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 0x7f5fb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:13.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.920+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fc4040fe0 con 0x7f5fd00f47c0 2023-11-15T05:57:13.924 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.920+0000 7f5fce7fc700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 0x7f5fb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:13.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.920+0000 7f5fce7fc700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 0x7f5fb8032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fc00096a0 tx=0x7f5fc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:13.925 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:13.924+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fc401d7f0 con 0x7f5fd00f47c0 2023-11-15T05:57:14.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.129+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fd0108130 con 0x7f5fd00f47c0 2023-11-15T05:57:14.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.130+0000 7f5fb77fe700 1 -- 172.21.15.137:0/968792078 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5fc4010230 con 0x7f5fd00f47c0 2023-11-15T05:57:14.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:14.131 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:14.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 msgr2=0x7f5fb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:14.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 0x7f5fb8032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f5fc00096a0 tx=0x7f5fc0006b40).stop 2023-11-15T05:57:14.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 msgr2=0x7f5fd0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:14.134 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5fc4013510 tx=0x7f5fc400b040).stop 2023-11-15T05:57:14.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 shutdown_connections 2023-11-15T05:57:14.135 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5fb802ff60 0x7f5fb8032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 --2- 172.21.15.137:0/968792078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fd00f47c0 0x7f5fd0106700 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 >> 172.21.15.137:0/968792078 conn(0x7f5fd00f02e0 msgr2=0x7f5fd00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 shutdown_connections 2023-11-15T05:57:14.137 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:14.131+0000 7f5fd5356700 1 -- 172.21.15.137:0/968792078 wait complete. 2023-11-15T05:57:14.137 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:14.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:12.987254+0000 mgr.smithi003.cldzke (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:15.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:14.130183+0000 mon.smithi003 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.137:0/968792078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:15.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:15.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:14.987598+0000 mgr.smithi003.cldzke (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:16.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 -- 172.21.15.137:0/1806353385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 msgr2=0x7f6e3c0f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:16.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1806353385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c0f6d80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6e2c009fb0 tx=0x7f6e2c009b30).stop 2023-11-15T05:57:16.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 -- 172.21.15.137:0/1806353385 shutdown_connections 2023-11-15T05:57:16.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1806353385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c0f6d80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 -- 172.21.15.137:0/1806353385 >> 172.21.15.137:0/1806353385 conn(0x7f6e3c0f0260 msgr2=0x7f6e3c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 -- 172.21.15.137:0/1806353385 shutdown_connections 2023-11-15T05:57:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.722+0000 7f6e414b1700 1 -- 172.21.15.137:0/1806353385 wait complete. 2023-11-15T05:57:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e414b1700 1 Processor -- start 2023-11-15T05:57:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e414b1700 1 -- start start 2023-11-15T05:57:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e414b1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e414b1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c0f52f0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e3affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e3affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:56588/0 (socket says 172.21.15.137:56588) 2023-11-15T05:57:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.723+0000 7f6e3affd700 1 -- 172.21.15.137:0/1440029523 learned_addr learned my addr 172.21.15.137:0/1440029523 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:16.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e3affd700 1 -- 172.21.15.137:0/1440029523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c005040 con 0x7f6e3c0f6960 2023-11-15T05:57:16.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e3affd700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6e2c013510 tx=0x7f6e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:16.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2c009320 con 0x7f6e3c0f6960 2023-11-15T05:57:16.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c0f54f0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e2c013f00 con 0x7f6e3c0f6960 2023-11-15T05:57:16.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2c014d90 con 0x7f6e3c0f6960 2023-11-15T05:57:16.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.724+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3c109aa0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e2c014420 con 0x7f6e3c0f6960 2023-11-15T05:57:16.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3c045ad0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e237fe700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 0x7f6e24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:16.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e2c041070 con 0x7f6e3c0f6960 2023-11-15T05:57:16.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e3a7fc700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 0x7f6e24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:16.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.725+0000 7f6e3a7fc700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 0x7f6e24032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6e300096a0 tx=0x7f6e30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:16.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.730+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e2c06d2b0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.937+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e3c10a2c0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.937+0000 7f6e237fe700 1 -- 172.21.15.137:0/1440029523 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6e2c0095f0 con 0x7f6e3c0f6960 2023-11-15T05:57:16.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:16.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:16.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 msgr2=0x7f6e24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:16.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 0x7f6e24032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6e300096a0 tx=0x7f6e30006b40).stop 2023-11-15T05:57:16.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 msgr2=0x7f6e3c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:16.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6e2c013510 tx=0x7f6e2c00b040).stop 2023-11-15T05:57:16.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 shutdown_connections 2023-11-15T05:57:16.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f6e2402ff60 0x7f6e24032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:16.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 --2- 172.21.15.137:0/1440029523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e3c0f6960 0x7f6e3c1093f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:16.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 >> 172.21.15.137:0/1440029523 conn(0x7f6e3c0f0260 msgr2=0x7f6e3c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:16.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 shutdown_connections 2023-11-15T05:57:16.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:16.939+0000 7f6e414b1700 1 -- 172.21.15.137:0/1440029523 wait complete. 2023-11-15T05:57:16.943 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:17.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:17 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:16.937751+0000 mon.smithi003 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.137:0/1440029523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:18.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:18.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:18.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:16.987926+0000 mgr.smithi003.cldzke (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:19.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 -- 172.21.15.137:0/468210471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 msgr2=0x7fb7240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:19.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/468210471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7240f90f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb718009fb0 tx=0x7fb718009b30).stop 2023-11-15T05:57:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 -- 172.21.15.137:0/468210471 shutdown_connections 2023-11-15T05:57:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/468210471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7240f90f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 -- 172.21.15.137:0/468210471 >> 172.21.15.137:0/468210471 conn(0x7fb7240f0b70 msgr2=0x7fb7240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 -- 172.21.15.137:0/468210471 shutdown_connections 2023-11-15T05:57:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.475+0000 7fb72a2f7700 1 -- 172.21.15.137:0/468210471 wait complete. 2023-11-15T05:57:19.478 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb72a2f7700 1 Processor -- start 2023-11-15T05:57:19.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb72a2f7700 1 -- start start 2023-11-15T05:57:19.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb72a2f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:19.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb72a2f7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb724106bd0 con 0x7fb724060280 2023-11-15T05:57:19.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb723fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:19.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb723fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45338/0 (socket says 172.21.15.137:45338) 2023-11-15T05:57:19.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.476+0000 7fb723fff700 1 -- 172.21.15.137:0/2660710864 learned_addr learned my addr 172.21.15.137:0/2660710864 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb723fff700 1 -- 172.21.15.137:0/2660710864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718005040 con 0x7fb724060280 2023-11-15T05:57:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb723fff700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb718013510 tx=0x7fb71800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb718009320 con 0x7fb724060280 2023-11-15T05:57:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb724106dd0 con 0x7fb724060280 2023-11-15T05:57:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb718013f00 con 0x7fb724060280 2023-11-15T05:57:19.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb718014dc0 con 0x7fb724060280 2023-11-15T05:57:19.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.477+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb724107260 con 0x7fb724060280 2023-11-15T05:57:19.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.478+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb718014420 con 0x7fb724060280 2023-11-15T05:57:19.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.478+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb724045ad0 con 0x7fb724060280 2023-11-15T05:57:19.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.478+0000 7fb720ff9700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:19.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.478+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb718041110 con 0x7fb724060280 2023-11-15T05:57:19.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.478+0000 7fb7237fe700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:19.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.479+0000 7fb7237fe700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb7140096a0 tx=0x7fb714006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:19.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.483+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7180095f0 con 0x7fb724060280 2023-11-15T05:57:19.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.688+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb724108110 con 0x7fb724060280 2023-11-15T05:57:19.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.689+0000 7fb720ff9700 1 -- 172.21.15.137:0/2660710864 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb71801cb10 con 0x7fb724060280 2023-11-15T05:57:19.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:19.690 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:19.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 msgr2=0x7fb70c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:19.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 0x7fb70c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb7140096a0 tx=0x7fb714006b40).stop 2023-11-15T05:57:19.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 msgr2=0x7fb7241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:19.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb718013510 tx=0x7fb71800b040).stop 2023-11-15T05:57:19.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 shutdown_connections 2023-11-15T05:57:19.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb70c02ff60 0x7fb70c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:19.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 --2- 172.21.15.137:0/2660710864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb724060280 0x7fb7241066e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:19.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.690+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 >> 172.21.15.137:0/2660710864 conn(0x7fb7240f0b70 msgr2=0x7fb7240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:19.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.691+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 shutdown_connections 2023-11-15T05:57:19.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:19.691+0000 7fb72a2f7700 1 -- 172.21.15.137:0/2660710864 wait complete. 2023-11-15T05:57:19.694 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:20.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:18.988230+0000 mgr.smithi003.cldzke (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:19.689329+0000 mon.smithi003 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.137:0/2660710864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:21.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:21.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:21.934 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:20.988599+0000 mgr.smithi003.cldzke (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:22.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 -- 172.21.15.137:0/3124550672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 msgr2=0x7f2a200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:22.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 --2- 172.21.15.137:0/3124550672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a200f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f2a10009fb0 tx=0x7f2a10009b30).stop 2023-11-15T05:57:22.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 -- 172.21.15.137:0/3124550672 shutdown_connections 2023-11-15T05:57:22.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 --2- 172.21.15.137:0/3124550672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a200f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:22.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 -- 172.21.15.137:0/3124550672 >> 172.21.15.137:0/3124550672 conn(0x7f2a200f0b70 msgr2=0x7f2a200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:22.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 -- 172.21.15.137:0/3124550672 shutdown_connections 2023-11-15T05:57:22.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.225+0000 7f2a267a3700 1 -- 172.21.15.137:0/3124550672 wait complete. 2023-11-15T05:57:22.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a267a3700 1 Processor -- start 2023-11-15T05:57:22.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a267a3700 1 -- start start 2023-11-15T05:57:22.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a267a3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:22.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a267a3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a20108e10 con 0x7f2a200f5410 2023-11-15T05:57:22.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a1ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:22.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a1ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45348/0 (socket says 172.21.15.137:45348) 2023-11-15T05:57:22.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.226+0000 7f2a1ffff700 1 -- 172.21.15.137:0/1498161157 learned_addr learned my addr 172.21.15.137:0/1498161157 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:22.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a1ffff700 1 -- 172.21.15.137:0/1498161157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10005040 con 0x7f2a200f5410 2023-11-15T05:57:22.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a1ffff700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2a10013510 tx=0x7f2a1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:22.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a10009320 con 0x7f2a200f5410 2023-11-15T05:57:22.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a20109010 con 0x7f2a200f5410 2023-11-15T05:57:22.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a10013f00 con 0x7f2a200f5410 2023-11-15T05:57:22.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a10014d90 con 0x7f2a200f5410 2023-11-15T05:57:22.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.227+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a201094a0 con 0x7f2a200f5410 2023-11-15T05:57:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.228+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a100095f0 con 0x7f2a200f5410 2023-11-15T05:57:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.228+0000 7f2a1cff9700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 0x7f2a0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.228+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a10040fe0 con 0x7f2a200f5410 2023-11-15T05:57:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.228+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a20045ad0 con 0x7f2a200f5410 2023-11-15T05:57:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.228+0000 7f2a1f7fe700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 0x7f2a0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:22.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.229+0000 7f2a1f7fe700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 0x7f2a0c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2a080096a0 tx=0x7f2a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:22.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.233+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a1001dba0 con 0x7f2a200f5410 2023-11-15T05:57:22.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.438+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a2010a1e0 con 0x7f2a200f5410 2023-11-15T05:57:22.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.439+0000 7f2a1cff9700 1 -- 172.21.15.137:0/1498161157 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2a10010230 con 0x7f2a200f5410 2023-11-15T05:57:22.439 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:22.440 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:22.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 msgr2=0x7f2a0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 0x7f2a0c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2a080096a0 tx=0x7f2a08006b40).stop 2023-11-15T05:57:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 msgr2=0x7f2a20108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2a10013510 tx=0x7f2a1000b040).stop 2023-11-15T05:57:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 shutdown_connections 2023-11-15T05:57:22.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a0c02ff60 0x7f2a0c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:22.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 --2- 172.21.15.137:0/1498161157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2a200f5410 0x7f2a20108920 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:22.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 >> 172.21.15.137:0/1498161157 conn(0x7f2a200f0b70 msgr2=0x7f2a2005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:22.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 shutdown_connections 2023-11-15T05:57:22.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:22.440+0000 7f2a267a3700 1 -- 172.21.15.137:0/1498161157 wait complete. 2023-11-15T05:57:22.443 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:22.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:22.439069+0000 mon.smithi003 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.137:0/1498161157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:23.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:23.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:23.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:22.988837+0000 mgr.smithi003.cldzke (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:24.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 -- 172.21.15.137:0/777708689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 msgr2=0x7f45840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:24.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 --2- 172.21.15.137:0/777708689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f45840f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4580009fb0 tx=0x7f4580009b30).stop 2023-11-15T05:57:24.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 -- 172.21.15.137:0/777708689 shutdown_connections 2023-11-15T05:57:24.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 --2- 172.21.15.137:0/777708689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f45840f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:24.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 -- 172.21.15.137:0/777708689 >> 172.21.15.137:0/777708689 conn(0x7f45840f0b70 msgr2=0x7f45840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:24.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 -- 172.21.15.137:0/777708689 shutdown_connections 2023-11-15T05:57:24.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.983+0000 7f458bdcd700 1 -- 172.21.15.137:0/777708689 wait complete. 2023-11-15T05:57:24.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f458bdcd700 1 Processor -- start 2023-11-15T05:57:24.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f458bdcd700 1 -- start start 2023-11-15T05:57:24.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f458bdcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:24.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f458bdcd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584106c10 con 0x7f45840f5410 2023-11-15T05:57:24.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f4589b69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:24.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f4589b69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45364/0 (socket says 172.21.15.137:45364) 2023-11-15T05:57:24.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.984+0000 7f4589b69700 1 -- 172.21.15.137:0/3704997700 learned_addr learned my addr 172.21.15.137:0/3704997700 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:24.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f4589b69700 1 -- 172.21.15.137:0/3704997700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4580005040 con 0x7f45840f5410 2023-11-15T05:57:24.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f4589b69700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4580013510 tx=0x7f458000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:24.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4580009320 con 0x7f45840f5410 2023-11-15T05:57:24.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4584106e10 con 0x7f45840f5410 2023-11-15T05:57:24.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4580013f00 con 0x7f45840f5410 2023-11-15T05:57:24.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4580014d90 con 0x7f45840f5410 2023-11-15T05:57:24.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.985+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45841072a0 con 0x7f45840f5410 2023-11-15T05:57:24.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.986+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45800095f0 con 0x7f45840f5410 2023-11-15T05:57:24.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.986+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4584045ad0 con 0x7f45840f5410 2023-11-15T05:57:24.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.986+0000 7f457a7fc700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 0x7f4570032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:24.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.986+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4580041080 con 0x7f45840f5410 2023-11-15T05:57:24.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.986+0000 7f4589368700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 0x7f4570032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:24.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.987+0000 7f4589368700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 0x7f4570032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f45740096a0 tx=0x7f4574006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:24.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:24.991+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4580014420 con 0x7f45840f5410 2023-11-15T05:57:25.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.199+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45841080f0 con 0x7f45840f5410 2023-11-15T05:57:25.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.199+0000 7f457a7fc700 1 -- 172.21.15.137:0/3704997700 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4580010230 con 0x7f45840f5410 2023-11-15T05:57:25.200 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:25.200 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:25.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 msgr2=0x7f4570032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:25.202 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 0x7f4570032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f45740096a0 tx=0x7f4574006b40).stop 2023-11-15T05:57:25.202 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 msgr2=0x7f4584106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:25.202 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4580013510 tx=0x7f458000b040).stop 2023-11-15T05:57:25.202 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 shutdown_connections 2023-11-15T05:57:25.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f457002ff60 0x7f4570032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:25.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 --2- 172.21.15.137:0/3704997700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f45840f5410 0x7f4584106720 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:25.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 >> 172.21.15.137:0/3704997700 conn(0x7f45840f0b70 msgr2=0x7f458405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:25.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 shutdown_connections 2023-11-15T05:57:25.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:25.201+0000 7f458bdcd700 1 -- 172.21.15.137:0/3704997700 wait complete. 2023-11-15T05:57:25.204 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:24.989075+0000 mgr.smithi003.cldzke (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:25.199641+0000 mon.smithi003 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.137:0/3704997700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:26.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:26.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:27.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 -- 172.21.15.137:0/1155175642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 msgr2=0x7f3b480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:27.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/1155175642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b480f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3b30009fb0 tx=0x7f3b30009b30).stop 2023-11-15T05:57:27.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 -- 172.21.15.137:0/1155175642 shutdown_connections 2023-11-15T05:57:27.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/1155175642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b480f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:27.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 -- 172.21.15.137:0/1155175642 >> 172.21.15.137:0/1155175642 conn(0x7f3b480f0b70 msgr2=0x7f3b480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 -- 172.21.15.137:0/1155175642 shutdown_connections 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.861+0000 7f3b4cae9700 1 -- 172.21.15.137:0/1155175642 wait complete. 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4cae9700 1 Processor -- start 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4cae9700 1 -- start start 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4cae9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:27.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4cae9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b48060220 con 0x7f3b480f5410 2023-11-15T05:57:27.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:27.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:45376/0 (socket says 172.21.15.137:45376) 2023-11-15T05:57:27.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4659c700 1 -- 172.21.15.137:0/4076886877 learned_addr learned my addr 172.21.15.137:0/4076886877 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:27.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4659c700 1 -- 172.21.15.137:0/4076886877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30005040 con 0x7f3b480f5410 2023-11-15T05:57:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.862+0000 7f3b4659c700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3b30013510 tx=0x7f3b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b30009320 con 0x7f3b480f5410 2023-11-15T05:57:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b48060420 con 0x7f3b480f5410 2023-11-15T05:57:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b30013f00 con 0x7f3b480f5410 2023-11-15T05:57:27.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b30014d90 con 0x7f3b480f5410 2023-11-15T05:57:27.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b48109b00 con 0x7f3b480f5410 2023-11-15T05:57:27.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b30014420 con 0x7f3b480f5410 2023-11-15T05:57:27.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.863+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b48059100 con 0x7f3b480f5410 2023-11-15T05:57:27.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.864+0000 7f3b3effd700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 0x7f3b34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:27.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.864+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b30040fb0 con 0x7f3b480f5410 2023-11-15T05:57:27.869 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.864+0000 7f3b45d9b700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 0x7f3b34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:27.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.864+0000 7f3b45d9b700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 0x7f3b34032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3b380096a0 tx=0x7f3b38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:27.870 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:27.868+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b300095f0 con 0x7f3b480f5410 2023-11-15T05:57:28.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.072+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b48045ad0 con 0x7f3b480f5410 2023-11-15T05:57:28.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.072+0000 7f3b3effd700 1 -- 172.21.15.137:0/4076886877 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3b3001db70 con 0x7f3b480f5410 2023-11-15T05:57:28.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:28.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:28.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 msgr2=0x7f3b34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:28.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 0x7f3b34032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3b380096a0 tx=0x7f3b38006b40).stop 2023-11-15T05:57:28.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 msgr2=0x7f3b48109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:28.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3b30013510 tx=0x7f3b3000b040).stop 2023-11-15T05:57:28.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 shutdown_connections 2023-11-15T05:57:28.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3b3402ff60 0x7f3b34032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:28.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 --2- 172.21.15.137:0/4076886877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b480f5410 0x7f3b48109450 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:28.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 >> 172.21.15.137:0/4076886877 conn(0x7f3b480f0b70 msgr2=0x7f3b480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:28.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 shutdown_connections 2023-11-15T05:57:28.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:28.074+0000 7f3b4cae9700 1 -- 172.21.15.137:0/4076886877 wait complete. 2023-11-15T05:57:28.077 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:26.989432+0000 mgr.smithi003.cldzke (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:29.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:29 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:28.072797+0000 mon.smithi003 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.137:0/4076886877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:29.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:29.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:30.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:28.989851+0000 mgr.smithi003.cldzke (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:30.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 -- 172.21.15.137:0/3577607704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 msgr2=0x7fa64c0f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:30.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 --2- 172.21.15.137:0/3577607704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c0f6bf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa640009fb0 tx=0x7fa640009b30).stop 2023-11-15T05:57:30.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 -- 172.21.15.137:0/3577607704 shutdown_connections 2023-11-15T05:57:30.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 --2- 172.21.15.137:0/3577607704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c0f6bf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:30.620 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 -- 172.21.15.137:0/3577607704 >> 172.21.15.137:0/3577607704 conn(0x7fa64c0effc0 msgr2=0x7fa64c0f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:30.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 -- 172.21.15.137:0/3577607704 shutdown_connections 2023-11-15T05:57:30.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 -- 172.21.15.137:0/3577607704 wait complete. 2023-11-15T05:57:30.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.617+0000 7fa651def700 1 Processor -- start 2023-11-15T05:57:30.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa651def700 1 -- start start 2023-11-15T05:57:30.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa651def700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:30.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa651def700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c106930 con 0x7fa64c0f67d0 2023-11-15T05:57:30.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa650ded700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:30.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa650ded700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:60858/0 (socket says 172.21.15.137:60858) 2023-11-15T05:57:30.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.618+0000 7fa650ded700 1 -- 172.21.15.137:0/3342976037 learned_addr learned my addr 172.21.15.137:0/3342976037 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:30.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa650ded700 1 -- 172.21.15.137:0/3342976037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640005040 con 0x7fa64c0f67d0 2023-11-15T05:57:30.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa650ded700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa6400089f0 tx=0x7fa64000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:30.628 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa640009320 con 0x7fa64c0f67d0 2023-11-15T05:57:30.628 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c106b30 con 0x7fa64c0f67d0 2023-11-15T05:57:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6400136c0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa64001d380 con 0x7fa64c0f67d0 2023-11-15T05:57:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.619+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c106fc0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.620+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa64001d4e0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.620+0000 7fa6497fa700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.620+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6400410c0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.620+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c045ad0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.620+0000 7fa64bfff700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:30.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.621+0000 7fa64bfff700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 0x7fa63c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:30.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.625+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6400263a0 con 0x7fa64c0f67d0 2023-11-15T05:57:30.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.835+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa64c107d10 con 0x7fa64c0f67d0 2023-11-15T05:57:30.836 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.836+0000 7fa6497fa700 1 -- 172.21.15.137:0/3342976037 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa640012330 con 0x7fa64c0f67d0 2023-11-15T05:57:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:30.837 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:30.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 msgr2=0x7fa63c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:30.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 0x7fa63c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa6380096a0 tx=0x7fa638006b40).stop 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 msgr2=0x7fa64c106440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa6400089f0 tx=0x7fa64000b040).stop 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 shutdown_connections 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 --2- 172.21.15.137:0/3342976037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64c0f67d0 0x7fa64c106440 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:30.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 >> 172.21.15.137:0/3342976037 conn(0x7fa64c0effc0 msgr2=0x7fa64c0f0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:30.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 shutdown_connections 2023-11-15T05:57:30.840 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:30.838+0000 7fa651def700 1 -- 172.21.15.137:0/3342976037 wait complete. 2023-11-15T05:57:30.840 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:31.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:30.836438+0000 mon.smithi003 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.137:0/3342976037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:30.990152+0000 mgr.smithi003.cldzke (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:32.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:32.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 -- 172.21.15.137:0/396776468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 msgr2=0x7f1fe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:33.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 --2- 172.21.15.137:0/396776468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe40f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1fd8009fb0 tx=0x7f1fd8009b30).stop 2023-11-15T05:57:33.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 -- 172.21.15.137:0/396776468 shutdown_connections 2023-11-15T05:57:33.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 --2- 172.21.15.137:0/396776468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe40f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:33.408 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 -- 172.21.15.137:0/396776468 >> 172.21.15.137:0/396776468 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 -- 172.21.15.137:0/396776468 shutdown_connections 2023-11-15T05:57:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.404+0000 7f1feac82700 1 -- 172.21.15.137:0/396776468 wait complete. 2023-11-15T05:57:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1feac82700 1 Processor -- start 2023-11-15T05:57:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1feac82700 1 -- start start 2023-11-15T05:57:33.409 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1feac82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1feac82700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4060280 con 0x7f1fe40f5410 2023-11-15T05:57:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1fe8a1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1fe8a1e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:60884/0 (socket says 172.21.15.137:60884) 2023-11-15T05:57:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1fe8a1e700 1 -- 172.21.15.137:0/829989706 learned_addr learned my addr 172.21.15.137:0/829989706 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:33.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.405+0000 7f1fe8a1e700 1 -- 172.21.15.137:0/829989706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8005040 con 0x7f1fe40f5410 2023-11-15T05:57:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1fe8a1e700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1fd8013510 tx=0x7f1fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fd8009320 con 0x7f1fe40f5410 2023-11-15T05:57:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe4060480 con 0x7f1fe40f5410 2023-11-15T05:57:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fd8013f00 con 0x7f1fe40f5410 2023-11-15T05:57:33.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fd8014d90 con 0x7f1fe40f5410 2023-11-15T05:57:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.406+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe4109b20 con 0x7f1fe40f5410 2023-11-15T05:57:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.407+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fd80095f0 con 0x7f1fe40f5410 2023-11-15T05:57:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.407+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe4045ad0 con 0x7f1fe40f5410 2023-11-15T05:57:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.407+0000 7f1fe17fa700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 0x7f1fd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:33.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.407+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fd8040fe0 con 0x7f1fe40f5410 2023-11-15T05:57:33.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.407+0000 7f1fe3fff700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 0x7f1fd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:33.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.408+0000 7f1fe3fff700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 0x7f1fd4032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1fd00096a0 tx=0x7f1fd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:33.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.412+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fd801d7f0 con 0x7f1fe40f5410 2023-11-15T05:57:33.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.617+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fe410a340 con 0x7f1fe40f5410 2023-11-15T05:57:33.619 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.618+0000 7f1fe17fa700 1 -- 172.21.15.137:0/829989706 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1fd8010230 con 0x7f1fe40f5410 2023-11-15T05:57:33.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:33.620 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:33.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.619+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 msgr2=0x7f1fd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:33.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.619+0000 7f1feac82700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 0x7f1fd4032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1fd00096a0 tx=0x7f1fd0006b40).stop 2023-11-15T05:57:33.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.619+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 msgr2=0x7f1fe41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.619+0000 7f1feac82700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1fd8013510 tx=0x7f1fd800b040).stop 2023-11-15T05:57:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 shutdown_connections 2023-11-15T05:57:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1fd402ff60 0x7f1fd4032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 --2- 172.21.15.137:0/829989706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fe40f5410 0x7f1fe41093f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 >> 172.21.15.137:0/829989706 conn(0x7f1fe40f0b70 msgr2=0x7f1fe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 shutdown_connections 2023-11-15T05:57:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:33.620+0000 7f1feac82700 1 -- 172.21.15.137:0/829989706 wait complete. 2023-11-15T05:57:33.623 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:32.990476+0000 mgr.smithi003.cldzke (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:33.619508+0000 mon.smithi003 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.137:0/829989706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:35.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:35.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:36.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 -- 172.21.15.137:0/3281381642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 msgr2=0x7f27ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:36.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 --2- 172.21.15.137:0/3281381642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac0f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f27a0009fb0 tx=0x7f27a0009b30).stop 2023-11-15T05:57:36.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 -- 172.21.15.137:0/3281381642 shutdown_connections 2023-11-15T05:57:36.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 --2- 172.21.15.137:0/3281381642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac0f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:36.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 -- 172.21.15.137:0/3281381642 >> 172.21.15.137:0/3281381642 conn(0x7f27ac0f0b70 msgr2=0x7f27ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:36.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 -- 172.21.15.137:0/3281381642 shutdown_connections 2023-11-15T05:57:36.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.213+0000 7f27b181d700 1 -- 172.21.15.137:0/3281381642 wait complete. 2023-11-15T05:57:36.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27b181d700 1 Processor -- start 2023-11-15T05:57:36.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27b181d700 1 -- start start 2023-11-15T05:57:36.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27b181d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:36.218 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27b181d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac106c10 con 0x7f27ac0f5410 2023-11-15T05:57:36.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27aaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:36.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27aaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:60908/0 (socket says 172.21.15.137:60908) 2023-11-15T05:57:36.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27aaffd700 1 -- 172.21.15.137:0/3348012749 learned_addr learned my addr 172.21.15.137:0/3348012749 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:36.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.214+0000 7f27aaffd700 1 -- 172.21.15.137:0/3348012749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0005040 con 0x7f27ac0f5410 2023-11-15T05:57:36.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27aaffd700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f27a0014000 tx=0x7f27a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:36.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a00095f0 con 0x7f27ac0f5410 2023-11-15T05:57:36.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac106e10 con 0x7f27ac0f5410 2023-11-15T05:57:36.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27a0007e70 con 0x7f27ac0f5410 2023-11-15T05:57:36.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a0014ea0 con 0x7f27ac0f5410 2023-11-15T05:57:36.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.215+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac1072a0 con 0x7f27ac0f5410 2023-11-15T05:57:36.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac045ad0 con 0x7f27ac0f5410 2023-11-15T05:57:36.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27a0025070 con 0x7f27ac0f5410 2023-11-15T05:57:36.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27937fe700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 0x7f2794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:36.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27a0040d10 con 0x7f27ac0f5410 2023-11-15T05:57:36.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27aa7fc700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 0x7f2794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:36.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.216+0000 7f27aa7fc700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 0x7f2794032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f279c0096a0 tx=0x7f279c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:36.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.221+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27a0025950 con 0x7f27ac0f5410 2023-11-15T05:57:36.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.428+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27ac1080f0 con 0x7f27ac0f5410 2023-11-15T05:57:36.430 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.428+0000 7f27937fe700 1 -- 172.21.15.137:0/3348012749 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f27a0025950 con 0x7f27ac0f5410 2023-11-15T05:57:36.431 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:36.431 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:36.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:34.990915+0000 mgr.smithi003.cldzke (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:36.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:35.112216+0000 mon.smithi003 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:57:36.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:35.115520+0000 mon.smithi003 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:57:36.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 msgr2=0x7f2794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:36.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 0x7f2794032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f279c0096a0 tx=0x7f279c006b40).stop 2023-11-15T05:57:36.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 msgr2=0x7f27ac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:36.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f27a0014000 tx=0x7f27a000b040).stop 2023-11-15T05:57:36.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 shutdown_connections 2023-11-15T05:57:36.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f279402ff60 0x7f2794032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:36.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 --2- 172.21.15.137:0/3348012749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27ac0f5410 0x7f27ac106720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:36.434 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 >> 172.21.15.137:0/3348012749 conn(0x7f27ac0f0b70 msgr2=0x7f27ac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:36.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 shutdown_connections 2023-11-15T05:57:36.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:36.430+0000 7f27b181d700 1 -- 172.21.15.137:0/3348012749 wait complete. 2023-11-15T05:57:36.435 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:37.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:37 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:36.430019+0000 mon.smithi003 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.137:0/3348012749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:37.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:37.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:36.991336+0000 mgr.smithi003.cldzke (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:39.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- 172.21.15.137:0/1168230912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 msgr2=0x7f54140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:39.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 --2- 172.21.15.137:0/1168230912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54140f55c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f5408009fb0 tx=0x7f5408009b30).stop 2023-11-15T05:57:39.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- 172.21.15.137:0/1168230912 shutdown_connections 2023-11-15T05:57:39.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 --2- 172.21.15.137:0/1168230912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54140f55c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:39.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- 172.21.15.137:0/1168230912 >> 172.21.15.137:0/1168230912 conn(0x7f54140f09a0 msgr2=0x7f54140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:39.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- 172.21.15.137:0/1168230912 shutdown_connections 2023-11-15T05:57:39.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- 172.21.15.137:0/1168230912 wait complete. 2023-11-15T05:57:39.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 Processor -- start 2023-11-15T05:57:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.007+0000 7f541359e700 1 -- start start 2023-11-15T05:57:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141058a0 con 0x7f54140f51a0 2023-11-15T05:57:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:39.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:60912/0 (socket says 172.21.15.137:60912) 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541259c700 1 -- 172.21.15.137:0/2895449980 learned_addr learned my addr 172.21.15.137:0/2895449980 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541259c700 1 -- 172.21.15.137:0/2895449980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408005040 con 0x7f54140f51a0 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541259c700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5408013510 tx=0x7f540800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5408009320 con 0x7f54140f51a0 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5414105aa0 con 0x7f54140f51a0 2023-11-15T05:57:39.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5408013f00 con 0x7f54140f51a0 2023-11-15T05:57:39.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.008+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5408014dc0 con 0x7f54140f51a0 2023-11-15T05:57:39.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.009+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5414105f30 con 0x7f54140f51a0 2023-11-15T05:57:39.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.009+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f540801d3d0 con 0x7f54140f51a0 2023-11-15T05:57:39.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.009+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8004fa0 con 0x7f54140f51a0 2023-11-15T05:57:39.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.010+0000 7f5402ffd700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 0x7f53fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:39.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.010+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5408041280 con 0x7f54140f51a0 2023-11-15T05:57:39.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.010+0000 7f5411d9b700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 0x7f53fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:39.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.010+0000 7f5411d9b700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 0x7f53fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f54040096a0 tx=0x7f5404006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:39.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.015+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5408012510 con 0x7f54140f51a0 2023-11-15T05:57:39.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.224+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53f8005670 con 0x7f54140f51a0 2023-11-15T05:57:39.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.225+0000 7f5402ffd700 1 -- 172.21.15.137:0/2895449980 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5408012330 con 0x7f54140f51a0 2023-11-15T05:57:39.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:39.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:39.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.226+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 msgr2=0x7f53fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:39.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.226+0000 7f541359e700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 0x7f53fc032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f54040096a0 tx=0x7f5404006b40).stop 2023-11-15T05:57:39.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.226+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 msgr2=0x7f54141053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:39.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.226+0000 7f541359e700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5408013510 tx=0x7f540800b040).stop 2023-11-15T05:57:39.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 shutdown_connections 2023-11-15T05:57:39.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f53fc02ff60 0x7f53fc032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:39.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 --2- 172.21.15.137:0/2895449980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54140f51a0 0x7f54141053b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:39.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 >> 172.21.15.137:0/2895449980 conn(0x7f54140f09a0 msgr2=0x7f54140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:39.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 shutdown_connections 2023-11-15T05:57:39.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:39.227+0000 7f541359e700 1 -- 172.21.15.137:0/2895449980 wait complete. 2023-11-15T05:57:39.230 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:40.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:38.991774+0000 mgr.smithi003.cldzke (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:39.226510+0000 mon.smithi003 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.137:0/2895449980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:40.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:40.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:41.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- 172.21.15.137:0/2612749230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 msgr2=0x7fe3500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:41.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 --2- 172.21.15.137:0/2612749230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3500f55c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe34c009fb0 tx=0x7fe34c009b30).stop 2023-11-15T05:57:41.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- 172.21.15.137:0/2612749230 shutdown_connections 2023-11-15T05:57:41.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 --2- 172.21.15.137:0/2612749230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3500f55c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:41.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- 172.21.15.137:0/2612749230 >> 172.21.15.137:0/2612749230 conn(0x7fe3500f09a0 msgr2=0x7fe3500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:41.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- 172.21.15.137:0/2612749230 shutdown_connections 2023-11-15T05:57:41.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- 172.21.15.137:0/2612749230 wait complete. 2023-11-15T05:57:41.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 Processor -- start 2023-11-15T05:57:41.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.775+0000 7fe35728a700 1 -- start start 2023-11-15T05:57:41.781 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe35728a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:41.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe35728a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3501058a0 con 0x7fe3500f51a0 2023-11-15T05:57:41.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe356288700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:41.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe356288700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55394/0 (socket says 172.21.15.137:55394) 2023-11-15T05:57:41.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe356288700 1 -- 172.21.15.137:0/3129812928 learned_addr learned my addr 172.21.15.137:0/3129812928 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:41.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe356288700 1 -- 172.21.15.137:0/3129812928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c005040 con 0x7fe3500f51a0 2023-11-15T05:57:41.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe356288700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe34c013510 tx=0x7fe34c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:41.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.776+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe34c009320 con 0x7fe3500f51a0 2023-11-15T05:57:41.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe350105aa0 con 0x7fe3500f51a0 2023-11-15T05:57:41.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe34c013f00 con 0x7fe3500f51a0 2023-11-15T05:57:41.784 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe34c014dc0 con 0x7fe3500f51a0 2023-11-15T05:57:41.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe350105f30 con 0x7fe3500f51a0 2023-11-15T05:57:41.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe34c0095f0 con 0x7fe3500f51a0 2023-11-15T05:57:41.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.777+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe350045ad0 con 0x7fe3500f51a0 2023-11-15T05:57:41.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.778+0000 7fe346ffd700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 0x7fe33c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:41.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.778+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe34c041000 con 0x7fe3500f51a0 2023-11-15T05:57:41.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.778+0000 7fe355a87700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 0x7fe33c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:41.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.778+0000 7fe355a87700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 0x7fe33c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe3400096a0 tx=0x7fe340006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:41.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.783+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe34c014420 con 0x7fe3500f51a0 2023-11-15T05:57:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:40.992209+0000 mgr.smithi003.cldzke (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:41.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.989+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe350106de0 con 0x7fe3500f51a0 2023-11-15T05:57:41.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.990+0000 7fe346ffd700 1 -- 172.21.15.137:0/3129812928 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe34c010230 con 0x7fe3500f51a0 2023-11-15T05:57:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:41.992 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 msgr2=0x7fe33c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 0x7fe33c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe3400096a0 tx=0x7fe340006b40).stop 2023-11-15T05:57:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 msgr2=0x7fe3501053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe34c013510 tx=0x7fe34c00b040).stop 2023-11-15T05:57:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 shutdown_connections 2023-11-15T05:57:41.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe33c02ff60 0x7fe33c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:41.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 --2- 172.21.15.137:0/3129812928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3500f51a0 0x7fe3501053b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:41.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 >> 172.21.15.137:0/3129812928 conn(0x7fe3500f09a0 msgr2=0x7fe3500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:41.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 shutdown_connections 2023-11-15T05:57:41.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:41.991+0000 7fe35728a700 1 -- 172.21.15.137:0/3129812928 wait complete. 2023-11-15T05:57:41.995 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:42.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:41.990964+0000 mon.smithi003 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.137:0/3129812928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:43.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:43.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:43.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:43 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:42.992623+0000 mgr.smithi003.cldzke (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:44.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.568+0000 7f979ca3f700 1 -- 172.21.15.137:0/3626018118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 msgr2=0x7f97980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:44.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.568+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3626018118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97980f66c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f978c009fb0 tx=0x7f978c009b30).stop 2023-11-15T05:57:44.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.568+0000 7f979ca3f700 1 -- 172.21.15.137:0/3626018118 shutdown_connections 2023-11-15T05:57:44.571 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.568+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3626018118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97980f66c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.568+0000 7f979ca3f700 1 -- 172.21.15.137:0/3626018118 >> 172.21.15.137:0/3626018118 conn(0x7f97980f1ac0 msgr2=0x7f97980f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 -- 172.21.15.137:0/3626018118 shutdown_connections 2023-11-15T05:57:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 -- 172.21.15.137:0/3626018118 wait complete. 2023-11-15T05:57:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 Processor -- start 2023-11-15T05:57:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 -- start start 2023-11-15T05:57:44.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:44.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.569+0000 7f979ca3f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9798106990 con 0x7f97980f62a0 2023-11-15T05:57:44.574 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f9796d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:44.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f9796d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55406/0 (socket says 172.21.15.137:55406) 2023-11-15T05:57:44.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f9796d9d700 1 -- 172.21.15.137:0/3919602980 learned_addr learned my addr 172.21.15.137:0/3919602980 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:44.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f9796d9d700 1 -- 172.21.15.137:0/3919602980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c005040 con 0x7f97980f62a0 2023-11-15T05:57:44.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f9796d9d700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f978c013510 tx=0x7f978c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:44.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.570+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f978c009320 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9798106b90 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f978c013f00 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f978c014dc0 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9798107020 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f978c01d3d0 con 0x7f97980f62a0 2023-11-15T05:57:44.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.571+0000 7f977f7fe700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 0x7f9780032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:44.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.572+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f978c041040 con 0x7f97980f62a0 2023-11-15T05:57:44.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.572+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9798045ad0 con 0x7f97980f62a0 2023-11-15T05:57:44.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.572+0000 7f979659c700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 0x7f9780032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:44.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.572+0000 7f979659c700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 0x7f9780032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f97880096a0 tx=0x7f9788006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:44.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.576+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f978c026490 con 0x7f97980f62a0 2023-11-15T05:57:44.782 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.781+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9798107e70 con 0x7f97980f62a0 2023-11-15T05:57:44.783 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.782+0000 7f977f7fe700 1 -- 172.21.15.137:0/3919602980 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f978c02b020 con 0x7f97980f62a0 2023-11-15T05:57:44.783 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:44.784 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:44.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 msgr2=0x7f9780032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:44.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 0x7f9780032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f97880096a0 tx=0x7f9788006b40).stop 2023-11-15T05:57:44.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 msgr2=0x7f97981064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:44.785 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f978c013510 tx=0x7f978c00b040).stop 2023-11-15T05:57:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 shutdown_connections 2023-11-15T05:57:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f978002ff60 0x7f9780032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 --2- 172.21.15.137:0/3919602980 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97980f62a0 0x7f97981064a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 >> 172.21.15.137:0/3919602980 conn(0x7f97980f1ac0 msgr2=0x7f97980f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 shutdown_connections 2023-11-15T05:57:44.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:44.783+0000 7f979ca3f700 1 -- 172.21.15.137:0/3919602980 wait complete. 2023-11-15T05:57:44.787 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:45.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:44.783045+0000 mon.smithi003 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.137:0/3919602980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:46.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:44.993041+0000 mgr.smithi003.cldzke (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:46.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:46.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:47.393 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 -- 172.21.15.137:0/1791743313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88300601e0 msgr2=0x7f88300f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 --2- 172.21.15.137:0/1791743313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88300601e0 0x7f88300f8f20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8824009fb0 tx=0x7f8824009b30).stop 2023-11-15T05:57:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 -- 172.21.15.137:0/1791743313 shutdown_connections 2023-11-15T05:57:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 --2- 172.21.15.137:0/1791743313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88300601e0 0x7f88300f8f20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 -- 172.21.15.137:0/1791743313 >> 172.21.15.137:0/1791743313 conn(0x7f88300f09a0 msgr2=0x7f88300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 -- 172.21.15.137:0/1791743313 shutdown_connections 2023-11-15T05:57:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.391+0000 7f8835dec700 1 -- 172.21.15.137:0/1791743313 wait complete. 2023-11-15T05:57:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8835dec700 1 Processor -- start 2023-11-15T05:57:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8835dec700 1 -- start start 2023-11-15T05:57:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8835dec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:47.396 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8835dec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8830106f00 con 0x7f88301065f0 2023-11-15T05:57:47.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8834dea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:47.397 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8834dea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55428/0 (socket says 172.21.15.137:55428) 2023-11-15T05:57:47.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.392+0000 7f8834dea700 1 -- 172.21.15.137:0/44933986 learned_addr learned my addr 172.21.15.137:0/44933986 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:47.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f8834dea700 1 -- 172.21.15.137:0/44933986 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8824005040 con 0x7f88301065f0 2023-11-15T05:57:47.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f8834dea700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8824009f80 tx=0x7f8824009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:47.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88240136c0 con 0x7f88301065f0 2023-11-15T05:57:47.398 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830107100 con 0x7f88301065f0 2023-11-15T05:57:47.399 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88240095f0 con 0x7f88301065f0 2023-11-15T05:57:47.401 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8830107c50 con 0x7f88301065f0 2023-11-15T05:57:47.401 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.393+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8824014420 con 0x7f88301065f0 2023-11-15T05:57:47.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.394+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8824025060 con 0x7f88301065f0 2023-11-15T05:57:47.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.394+0000 7f882d7fa700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 0x7f88200324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:47.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.394+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8824040f00 con 0x7f88301065f0 2023-11-15T05:57:47.402 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.394+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8830059100 con 0x7f88301065f0 2023-11-15T05:57:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.394+0000 7f882ffff700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 0x7f88200324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.395+0000 7f882ffff700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 0x7f88200324c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f881c0096a0 tx=0x7f881c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:47.403 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.399+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8824028440 con 0x7f88301065f0 2023-11-15T05:57:47.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.603+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8830108370 con 0x7f88301065f0 2023-11-15T05:57:47.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.604+0000 7f882d7fa700 1 -- 172.21.15.137:0/44933986 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f882401c340 con 0x7f88301065f0 2023-11-15T05:57:47.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:47.607 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:47.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.605+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 msgr2=0x7f88200324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:47.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.605+0000 7f8835dec700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 0x7f88200324c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f881c0096a0 tx=0x7f881c006b40).stop 2023-11-15T05:57:47.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.605+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 msgr2=0x7f8830106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:47.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.605+0000 7f8835dec700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8824009f80 tx=0x7f8824009b30).stop 2023-11-15T05:57:47.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 shutdown_connections 2023-11-15T05:57:47.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8820030000 0x7f88200324c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:47.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 --2- 172.21.15.137:0/44933986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88301065f0 0x7f8830106a10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:47.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 >> 172.21.15.137:0/44933986 conn(0x7f88300f09a0 msgr2=0x7f88300f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:47.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 shutdown_connections 2023-11-15T05:57:47.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:47.606+0000 7f8835dec700 1 -- 172.21.15.137:0/44933986 wait complete. 2023-11-15T05:57:47.610 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:48.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:46.993435+0000 mgr.smithi003.cldzke (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:48.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:47.605316+0000 mon.smithi003 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.137:0/44933986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:49.019 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:49.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:50.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/501174647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 msgr2=0x7f2d240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/501174647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d240f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2d20009fb0 tx=0x7f2d20009b30).stop 2023-11-15T05:57:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/501174647 shutdown_connections 2023-11-15T05:57:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/501174647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d240f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/501174647 >> 172.21.15.137:0/501174647 conn(0x7f2d240f1c70 msgr2=0x7f2d240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.220+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/501174647 shutdown_connections 2023-11-15T05:57:50.224 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.221+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/501174647 wait complete. 2023-11-15T05:57:50.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.221+0000 7f2d2c4e7700 1 Processor -- start 2023-11-15T05:57:50.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.221+0000 7f2d2c4e7700 1 -- start start 2023-11-15T05:57:50.226 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.221+0000 7f2d2c4e7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:50.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.221+0000 7f2d2c4e7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d24106c20 con 0x7f2d240f6510 2023-11-15T05:57:50.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.222+0000 7f2d2a283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:50.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.222+0000 7f2d2a283700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57276/0 (socket says 172.21.15.137:57276) 2023-11-15T05:57:50.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.222+0000 7f2d2a283700 1 -- 172.21.15.137:0/1029445507 learned_addr learned my addr 172.21.15.137:0/1029445507 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:50.227 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.222+0000 7f2d2a283700 1 -- 172.21.15.137:0/1029445507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d20005040 con 0x7f2d240f6510 2023-11-15T05:57:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.222+0000 7f2d2a283700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d20013510 tx=0x7f2d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.223+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d20009320 con 0x7f2d240f6510 2023-11-15T05:57:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.223+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d24106e20 con 0x7f2d240f6510 2023-11-15T05:57:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.223+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d20013f00 con 0x7f2d240f6510 2023-11-15T05:57:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.223+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d20014d90 con 0x7f2d240f6510 2023-11-15T05:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.223+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d241072b0 con 0x7f2d240f6510 2023-11-15T05:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d200095f0 con 0x7f2d240f6510 2023-11-15T05:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d24045ad0 con 0x7f2d240f6510 2023-11-15T05:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d1affd700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:50.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d20041000 con 0x7f2d240f6510 2023-11-15T05:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d29a82700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.224+0000 7f2d29a82700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 0x7f2d10032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2d140096a0 tx=0x7f2d14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:50.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.229+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d20014420 con 0x7f2d240f6510 2023-11-15T05:57:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:48.993824+0000 mgr.smithi003.cldzke (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:50.435 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.434+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d240fbf60 con 0x7f2d240f6510 2023-11-15T05:57:50.436 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.434+0000 7f2d1affd700 1 -- 172.21.15.137:0/1029445507 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2d2001d7f0 con 0x7f2d240f6510 2023-11-15T05:57:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:50.436 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:50.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 msgr2=0x7f2d10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:50.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 0x7f2d10032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2d140096a0 tx=0x7f2d14006b40).stop 2023-11-15T05:57:50.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 msgr2=0x7f2d24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:50.438 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d20013510 tx=0x7f2d2000b040).stop 2023-11-15T05:57:50.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 shutdown_connections 2023-11-15T05:57:50.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d1002ff60 0x7f2d10032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:50.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 --2- 172.21.15.137:0/1029445507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d240f6510 0x7f2d24106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:50.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 >> 172.21.15.137:0/1029445507 conn(0x7f2d240f1c70 msgr2=0x7f2d240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:50.439 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 shutdown_connections 2023-11-15T05:57:50.440 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:50.436+0000 7f2d2c4e7700 1 -- 172.21.15.137:0/1029445507 wait complete. 2023-11-15T05:57:50.440 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:51.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:50.435853+0000 mon.smithi003 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.137:0/1029445507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:51.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:51.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:52.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:50.994126+0000 mgr.smithi003.cldzke (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:53.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 -- 172.21.15.137:0/1190624669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 msgr2=0x7f25140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:53.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 --2- 172.21.15.137:0/1190624669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f25140f66c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2508009fb0 tx=0x7f2508009b30).stop 2023-11-15T05:57:53.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 -- 172.21.15.137:0/1190624669 shutdown_connections 2023-11-15T05:57:53.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 --2- 172.21.15.137:0/1190624669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f25140f66c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:53.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 -- 172.21.15.137:0/1190624669 >> 172.21.15.137:0/1190624669 conn(0x7f25140f1ac0 msgr2=0x7f25140f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:53.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 -- 172.21.15.137:0/1190624669 shutdown_connections 2023-11-15T05:57:53.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.039+0000 7f251b58f700 1 -- 172.21.15.137:0/1190624669 wait complete. 2023-11-15T05:57:53.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.040+0000 7f251b58f700 1 Processor -- start 2023-11-15T05:57:53.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.040+0000 7f251b58f700 1 -- start start 2023-11-15T05:57:53.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.040+0000 7f251b58f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:53.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.040+0000 7f251b58f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141069f0 con 0x7f25140f62a0 2023-11-15T05:57:53.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251a58d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:53.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251a58d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57294/0 (socket says 172.21.15.137:57294) 2023-11-15T05:57:53.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251a58d700 1 -- 172.21.15.137:0/261754955 learned_addr learned my addr 172.21.15.137:0/261754955 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:53.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251a58d700 1 -- 172.21.15.137:0/261754955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2508005040 con 0x7f25140f62a0 2023-11-15T05:57:53.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251a58d700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2508013510 tx=0x7f250800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:53.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2508009320 con 0x7f25140f62a0 2023-11-15T05:57:53.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2514106bf0 con 0x7f25140f62a0 2023-11-15T05:57:53.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2508013f00 con 0x7f25140f62a0 2023-11-15T05:57:53.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2508014dc0 con 0x7f25140f62a0 2023-11-15T05:57:53.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.041+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2514107080 con 0x7f25140f62a0 2023-11-15T05:57:53.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.042+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25080095f0 con 0x7f25140f62a0 2023-11-15T05:57:53.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.042+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2514045ad0 con 0x7f25140f62a0 2023-11-15T05:57:53.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.042+0000 7f2506ffd700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 0x7f2500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:53.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.042+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2508041020 con 0x7f25140f62a0 2023-11-15T05:57:53.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.042+0000 7f2519d8c700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 0x7f2500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:53.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.043+0000 7f2519d8c700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 0x7f2500032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f25100096a0 tx=0x7f2510006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:53.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.047+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2508014420 con 0x7f25140f62a0 2023-11-15T05:57:53.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.250+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2514107eb0 con 0x7f25140f62a0 2023-11-15T05:57:53.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.251+0000 7f2506ffd700 1 -- 172.21.15.137:0/261754955 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f250801c920 con 0x7f25140f62a0 2023-11-15T05:57:53.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:53.253 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:53.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 msgr2=0x7f2500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:53.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 0x7f2500032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f25100096a0 tx=0x7f2510006b40).stop 2023-11-15T05:57:53.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 msgr2=0x7f2514106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:53.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2508013510 tx=0x7f250800b040).stop 2023-11-15T05:57:53.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 shutdown_connections 2023-11-15T05:57:53.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f250002ff60 0x7f2500032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:53.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 --2- 172.21.15.137:0/261754955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25140f62a0 0x7f2514106500 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:53.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 >> 172.21.15.137:0/261754955 conn(0x7f25140f1ac0 msgr2=0x7f25140f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:53.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.252+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 shutdown_connections 2023-11-15T05:57:53.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:53.253+0000 7f251b58f700 1 -- 172.21.15.137:0/261754955 wait complete. 2023-11-15T05:57:53.256 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:54.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:52.994387+0000 mgr.smithi003.cldzke (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:54.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:53.252340+0000 mon.smithi003 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.137:0/261754955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:54.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:54.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:55.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/1780698925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 msgr2=0x7f7b340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:55.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/1780698925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b340f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f7b24009fb0 tx=0x7f7b24009b30).stop 2023-11-15T05:57:55.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/1780698925 shutdown_connections 2023-11-15T05:57:55.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/1780698925 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b340f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:55.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/1780698925 >> 172.21.15.137:0/1780698925 conn(0x7f7b340f1c70 msgr2=0x7f7b340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:55.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/1780698925 shutdown_connections 2023-11-15T05:57:55.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.882+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/1780698925 wait complete. 2023-11-15T05:57:55.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3c4c4700 1 Processor -- start 2023-11-15T05:57:55.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3c4c4700 1 -- start start 2023-11-15T05:57:55.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3c4c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:55.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3c4c4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b34106c20 con 0x7f7b340f6510 2023-11-15T05:57:55.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3a260700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:55.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3a260700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57308/0 (socket says 172.21.15.137:57308) 2023-11-15T05:57:55.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.883+0000 7f7b3a260700 1 -- 172.21.15.137:0/3104753829 learned_addr learned my addr 172.21.15.137:0/3104753829 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:55.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b3a260700 1 -- 172.21.15.137:0/3104753829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24005040 con 0x7f7b340f6510 2023-11-15T05:57:55.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b3a260700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f7b24013510 tx=0x7f7b2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:55.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b24009320 con 0x7f7b340f6510 2023-11-15T05:57:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b34106e20 con 0x7f7b340f6510 2023-11-15T05:57:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b341072b0 con 0x7f7b340f6510 2023-11-15T05:57:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b24013f00 con 0x7f7b340f6510 2023-11-15T05:57:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.884+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b24014d60 con 0x7f7b340f6510 2023-11-15T05:57:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.885+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b24014420 con 0x7f7b340f6510 2023-11-15T05:57:55.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.885+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1c004fa0 con 0x7f7b340f6510 2023-11-15T05:57:55.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.885+0000 7f7b2affd700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:55.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.885+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b240412e0 con 0x7f7b340f6510 2023-11-15T05:57:55.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.885+0000 7f7b39a5f700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:55.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.886+0000 7f7b39a5f700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 0x7f7b20032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7b300096a0 tx=0x7f7b30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:55.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:55.890+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b24012b20 con 0x7f7b340f6510 2023-11-15T05:57:56.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.093+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b1c005670 con 0x7f7b340f6510 2023-11-15T05:57:56.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.094+0000 7f7b2affd700 1 -- 172.21.15.137:0/3104753829 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7b24012b20 con 0x7f7b340f6510 2023-11-15T05:57:56.096 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:56.096 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:56.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.095+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 msgr2=0x7f7b20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:56.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.095+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 0x7f7b20032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7b300096a0 tx=0x7f7b30006b40).stop 2023-11-15T05:57:56.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 msgr2=0x7f7b34106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:56.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f7b24013510 tx=0x7f7b2400b040).stop 2023-11-15T05:57:56.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 shutdown_connections 2023-11-15T05:57:56.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7b2002ff60 0x7f7b20032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:56.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 --2- 172.21.15.137:0/3104753829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b340f6510 0x7f7b34106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:56.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 >> 172.21.15.137:0/3104753829 conn(0x7f7b340f1c70 msgr2=0x7f7b340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:56.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 shutdown_connections 2023-11-15T05:57:56.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:56.096+0000 7f7b3c4c4700 1 -- 172.21.15.137:0/3104753829 wait complete. 2023-11-15T05:57:56.099 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:54.994689+0000 mgr.smithi003.cldzke (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:57.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:57 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:56.095240+0000 mon.smithi003 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.137:0/3104753829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:57:57.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:57:57.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:57:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:56.994965+0000 mgr.smithi003.cldzke (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:57:58.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2653877580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 msgr2=0x7f2aa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:58.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2653877580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa40f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2a98009fb0 tx=0x7f2a98009b30).stop 2023-11-15T05:57:58.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2653877580 shutdown_connections 2023-11-15T05:57:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2653877580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa40f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2653877580 >> 172.21.15.137:0/2653877580 conn(0x7f2aa40f0b70 msgr2=0x7f2aa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2653877580 shutdown_connections 2023-11-15T05:57:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.668+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2653877580 wait complete. 2023-11-15T05:57:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa8dbd700 1 Processor -- start 2023-11-15T05:57:58.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa8dbd700 1 -- start start 2023-11-15T05:57:58.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa8dbd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:58.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa8dbd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4105b20 con 0x7f2aa40f5410 2023-11-15T05:57:58.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:58.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57328/0 (socket says 172.21.15.137:57328) 2023-11-15T05:57:58.674 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.669+0000 7f2aa259c700 1 -- 172.21.15.137:0/2938709271 learned_addr learned my addr 172.21.15.137:0/2938709271 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:57:58.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2aa259c700 1 -- 172.21.15.137:0/2938709271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98005040 con 0x7f2aa40f5410 2023-11-15T05:57:58.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2aa259c700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f2a98013510 tx=0x7f2a9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:58.678 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a98009320 con 0x7f2aa40f5410 2023-11-15T05:57:58.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa4105d20 con 0x7f2aa40f5410 2023-11-15T05:57:58.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a98013f00 con 0x7f2aa40f5410 2023-11-15T05:57:58.679 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a98014d90 con 0x7f2aa40f5410 2023-11-15T05:57:58.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.670+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa41061b0 con 0x7f2aa40f5410 2023-11-15T05:57:58.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.671+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a980263a0 con 0x7f2aa40f5410 2023-11-15T05:57:58.680 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.671+0000 7f2a92ffd700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 0x7f2a8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:57:58.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.671+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa4045ad0 con 0x7f2aa40f5410 2023-11-15T05:57:58.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.671+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a980410a0 con 0x7f2aa40f5410 2023-11-15T05:57:58.681 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.671+0000 7f2aa1d9b700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 0x7f2a8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:57:58.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.672+0000 7f2aa1d9b700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 0x7f2a8c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2a940096a0 tx=0x7f2a94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:57:58.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.676+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a98012920 con 0x7f2aa40f5410 2023-11-15T05:57:58.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.883+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2aa40faf50 con 0x7f2aa40f5410 2023-11-15T05:57:58.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.883+0000 7f2a92ffd700 1 -- 172.21.15.137:0/2938709271 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2a98012330 con 0x7f2aa40f5410 2023-11-15T05:57:58.886 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:57:58.886 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:57:58.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 msgr2=0x7f2a8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:58.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 0x7f2a8c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2a940096a0 tx=0x7f2a94006b40).stop 2023-11-15T05:57:58.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 msgr2=0x7f2aa4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:57:58.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f2a98013510 tx=0x7f2a9800b040).stop 2023-11-15T05:57:58.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 shutdown_connections 2023-11-15T05:57:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2a8c02ff60 0x7f2a8c032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 --2- 172.21.15.137:0/2938709271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2aa40f5410 0x7f2aa4105630 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:57:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 >> 172.21.15.137:0/2938709271 conn(0x7f2aa40f0b70 msgr2=0x7f2aa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:57:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 shutdown_connections 2023-11-15T05:57:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:57:58.885+0000 7f2aa8dbd700 1 -- 172.21.15.137:0/2938709271 wait complete. 2023-11-15T05:57:58.889 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:57:59.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:57:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:57:58.884737+0000 mon.smithi003 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.137:0/2938709271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:00.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:00.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:00.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:57:58.995243+0000 mgr.smithi003.cldzke (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:01.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 -- 172.21.15.137:0/2671491822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 msgr2=0x7f643c0f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:01.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2671491822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f4c00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f642c009fb0 tx=0x7f642c009b30).stop 2023-11-15T05:58:01.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 -- 172.21.15.137:0/2671491822 shutdown_connections 2023-11-15T05:58:01.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2671491822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f4c00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:01.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 -- 172.21.15.137:0/2671491822 >> 172.21.15.137:0/2671491822 conn(0x7f643c0f0260 msgr2=0x7f643c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:01.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 -- 172.21.15.137:0/2671491822 shutdown_connections 2023-11-15T05:58:01.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.477+0000 7f6443a9b700 1 -- 172.21.15.137:0/2671491822 wait complete. 2023-11-15T05:58:01.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6443a9b700 1 Processor -- start 2023-11-15T05:58:01.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6443a9b700 1 -- start start 2023-11-15T05:58:01.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6443a9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:01.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6443a9b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c0f5850 con 0x7f643c0f47e0 2023-11-15T05:58:01.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6441837700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:01.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6441837700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53894/0 (socket says 172.21.15.137:53894) 2023-11-15T05:58:01.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.478+0000 7f6441837700 1 -- 172.21.15.137:0/2888353561 learned_addr learned my addr 172.21.15.137:0/2888353561 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f6441837700 1 -- 172.21.15.137:0/2888353561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c005040 con 0x7f643c0f47e0 2023-11-15T05:58:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f6441837700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f642c009830 tx=0x7f642c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f642c009320 con 0x7f643c0f47e0 2023-11-15T05:58:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c0f59f0 con 0x7f643c0f47e0 2023-11-15T05:58:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f642c0136c0 con 0x7f643c0f47e0 2023-11-15T05:58:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c0f5e80 con 0x7f643c0f47e0 2023-11-15T05:58:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.479+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f642c01ee90 con 0x7f643c0f47e0 2023-11-15T05:58:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.480+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f642c01e420 con 0x7f643c0f47e0 2023-11-15T05:58:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.480+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c059100 con 0x7f643c0f47e0 2023-11-15T05:58:01.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.480+0000 7f64327fc700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 0x7f64280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:01.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.480+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f642c041010 con 0x7f643c0f47e0 2023-11-15T05:58:01.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.480+0000 7f6441036700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 0x7f64280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:01.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.481+0000 7f6441036700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 0x7f64280323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f64380096a0 tx=0x7f6438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:01.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.485+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f642c0095f0 con 0x7f643c0f47e0 2023-11-15T05:58:01.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.689+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f643c045ad0 con 0x7f643c0f47e0 2023-11-15T05:58:01.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.690+0000 7f64327fc700 1 -- 172.21.15.137:0/2888353561 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f642c01cb10 con 0x7f643c0f47e0 2023-11-15T05:58:01.691 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:01.692 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 msgr2=0x7f64280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 0x7f64280323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f64380096a0 tx=0x7f6438006b40).stop 2023-11-15T05:58:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 msgr2=0x7f643c0f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f642c009830 tx=0x7f642c00b040).stop 2023-11-15T05:58:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 shutdown_connections 2023-11-15T05:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f642802ff10 0x7f64280323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 --2- 172.21.15.137:0/2888353561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f643c0f47e0 0x7f643c0f73d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 >> 172.21.15.137:0/2888353561 conn(0x7f643c0f0260 msgr2=0x7f643c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 shutdown_connections 2023-11-15T05:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:01.691+0000 7f6443a9b700 1 -- 172.21.15.137:0/2888353561 wait complete. 2023-11-15T05:58:01.695 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:01.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:00.995498+0000 mgr.smithi003.cldzke (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:02.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:01.690972+0000 mon.smithi003 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.137:0/2888353561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:03.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:03.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:03.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:02.995862+0000 mgr.smithi003.cldzke (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:04.240 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- 172.21.15.137:0/881852749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 msgr2=0x7f7fd00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:04.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 --2- 172.21.15.137:0/881852749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd00f66c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f7fc0009fb0 tx=0x7f7fc0009b30).stop 2023-11-15T05:58:04.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- 172.21.15.137:0/881852749 shutdown_connections 2023-11-15T05:58:04.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 --2- 172.21.15.137:0/881852749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd00f66c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:04.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- 172.21.15.137:0/881852749 >> 172.21.15.137:0/881852749 conn(0x7f7fd00f1aa0 msgr2=0x7f7fd00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:04.241 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- 172.21.15.137:0/881852749 shutdown_connections 2023-11-15T05:58:04.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- 172.21.15.137:0/881852749 wait complete. 2023-11-15T05:58:04.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 Processor -- start 2023-11-15T05:58:04.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- start start 2023-11-15T05:58:04.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:04.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.238+0000 7f7fd5695700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0108ba0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:04.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53908/0 (socket says 172.21.15.137:53908) 2023-11-15T05:58:04.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fcffff700 1 -- 172.21.15.137:0/1021708169 learned_addr learned my addr 172.21.15.137:0/1021708169 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:04.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fcffff700 1 -- 172.21.15.137:0/1021708169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0005040 con 0x7f7fd00f62a0 2023-11-15T05:58:04.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fcffff700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7fc0013510 tx=0x7f7fc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:04.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fc0009320 con 0x7f7fd00f62a0 2023-11-15T05:58:04.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd0108da0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fc0013f00 con 0x7f7fd00f62a0 2023-11-15T05:58:04.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fc0014dc0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.239+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd0109230 con 0x7f7fd00f62a0 2023-11-15T05:58:04.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.240+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fc00095f0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.240+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd0045ad0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.240+0000 7f7fccff9700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 0x7f7fb8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:04.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.240+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fc00410e0 con 0x7f7fd00f62a0 2023-11-15T05:58:04.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.241+0000 7f7fcf7fe700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 0x7f7fb8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:04.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.241+0000 7f7fcf7fe700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 0x7f7fb8032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7fc40096a0 tx=0x7f7fc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:04.248 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.245+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fc0014420 con 0x7f7fd00f62a0 2023-11-15T05:58:04.450 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.449+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fd010a080 con 0x7f7fd00f62a0 2023-11-15T05:58:04.451 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.450+0000 7f7fccff9700 1 -- 172.21.15.137:0/1021708169 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7fc001c920 con 0x7f7fd00f62a0 2023-11-15T05:58:04.452 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:04.452 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:04.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 msgr2=0x7f7fb8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:04.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 0x7f7fb8032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f7fc40096a0 tx=0x7f7fc4006b40).stop 2023-11-15T05:58:04.453 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 msgr2=0x7f7fd01086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:04.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7fc0013510 tx=0x7f7fc000b040).stop 2023-11-15T05:58:04.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 shutdown_connections 2023-11-15T05:58:04.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7fb802ffb0 0x7f7fb8032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:04.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 --2- 172.21.15.137:0/1021708169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fd00f62a0 0x7f7fd01086b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:04.454 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 >> 172.21.15.137:0/1021708169 conn(0x7f7fd00f1aa0 msgr2=0x7f7fd00f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:04.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 shutdown_connections 2023-11-15T05:58:04.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:04.451+0000 7f7fd5695700 1 -- 172.21.15.137:0/1021708169 wait complete. 2023-11-15T05:58:04.455 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:04.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:04.451213+0000 mon.smithi003 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.137:0/1021708169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:05.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:05.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:05.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:04.996201+0000 mgr.smithi003.cldzke (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:07.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- 172.21.15.137:0/656611068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 msgr2=0x7f65fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:07.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 --2- 172.21.15.137:0/656611068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc0f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f65ec009fb0 tx=0x7f65ec009b30).stop 2023-11-15T05:58:07.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- 172.21.15.137:0/656611068 shutdown_connections 2023-11-15T05:58:07.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 --2- 172.21.15.137:0/656611068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc0f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:07.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- 172.21.15.137:0/656611068 >> 172.21.15.137:0/656611068 conn(0x7f65fc0f0b70 msgr2=0x7f65fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:07.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- 172.21.15.137:0/656611068 shutdown_connections 2023-11-15T05:58:07.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- 172.21.15.137:0/656611068 wait complete. 2023-11-15T05:58:07.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 Processor -- start 2023-11-15T05:58:07.005 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.001+0000 7f66038cd700 1 -- start start 2023-11-15T05:58:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f66038cd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f66038cd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc106bd0 con 0x7f65fc0f5410 2023-11-15T05:58:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f6601669700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f6601669700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53934/0 (socket says 172.21.15.137:53934) 2023-11-15T05:58:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f6601669700 1 -- 172.21.15.137:0/555951392 learned_addr learned my addr 172.21.15.137:0/555951392 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:07.007 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f6601669700 1 -- 172.21.15.137:0/555951392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec005040 con 0x7f65fc0f5410 2023-11-15T05:58:07.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f6601669700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f65ec013510 tx=0x7f65ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:07.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec009320 con 0x7f65fc0f5410 2023-11-15T05:58:07.008 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc106dd0 con 0x7f65fc0f5410 2023-11-15T05:58:07.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65ec013f00 con 0x7f65fc0f5410 2023-11-15T05:58:07.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.002+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65ec014d90 con 0x7f65fc0f5410 2023-11-15T05:58:07.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.003+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc107260 con 0x7f65fc0f5410 2023-11-15T05:58:07.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.003+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65ec0263a0 con 0x7f65fc0f5410 2023-11-15T05:58:07.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.004+0000 7f65f1ffb700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 0x7f65e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:07.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.004+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc045ad0 con 0x7f65fc0f5410 2023-11-15T05:58:07.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.004+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65ec041300 con 0x7f65fc0f5410 2023-11-15T05:58:07.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.004+0000 7f6600e68700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 0x7f65e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:07.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.004+0000 7f6600e68700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 0x7f65e8032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f65f80096a0 tx=0x7f65f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:07.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.009+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65ec014420 con 0x7f65fc0f5410 2023-11-15T05:58:07.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.215+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65fc108040 con 0x7f65fc0f5410 2023-11-15T05:58:07.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.216+0000 7f65f1ffb700 1 -- 172.21.15.137:0/555951392 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f65ec00f340 con 0x7f65fc0f5410 2023-11-15T05:58:07.218 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:07.218 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:07.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 msgr2=0x7f65e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:07.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 0x7f65e8032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f65f80096a0 tx=0x7f65f8006b40).stop 2023-11-15T05:58:07.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 msgr2=0x7f65fc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:07.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f65ec013510 tx=0x7f65ec00b040).stop 2023-11-15T05:58:07.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 shutdown_connections 2023-11-15T05:58:07.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f65e802ff60 0x7f65e8032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:07.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 --2- 172.21.15.137:0/555951392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f65fc0f5410 0x7f65fc1066e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:07.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.217+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 >> 172.21.15.137:0/555951392 conn(0x7f65fc0f0b70 msgr2=0x7f65fc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:07.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.218+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 shutdown_connections 2023-11-15T05:58:07.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:07.218+0000 7f66038cd700 1 -- 172.21.15.137:0/555951392 wait complete. 2023-11-15T05:58:07.223 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:06.996532+0000 mgr.smithi003.cldzke (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:07.217143+0000 mon.smithi003 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.137:0/555951392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:08.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:08.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:09.740 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 -- 172.21.15.137:0/2572925112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 msgr2=0x7f44200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:09.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 --2- 172.21.15.137:0/2572925112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f44200f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f4414009fb0 tx=0x7f4414009b30).stop 2023-11-15T05:58:09.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 -- 172.21.15.137:0/2572925112 shutdown_connections 2023-11-15T05:58:09.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 --2- 172.21.15.137:0/2572925112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f44200f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:09.741 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 -- 172.21.15.137:0/2572925112 >> 172.21.15.137:0/2572925112 conn(0x7f44200f0b70 msgr2=0x7f44200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:09.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 -- 172.21.15.137:0/2572925112 shutdown_connections 2023-11-15T05:58:09.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 -- 172.21.15.137:0/2572925112 wait complete. 2023-11-15T05:58:09.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.737+0000 7f4424af1700 1 Processor -- start 2023-11-15T05:58:09.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f4424af1700 1 -- start start 2023-11-15T05:58:09.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f4424af1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:09.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f4424af1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420060240 con 0x7f44200f5410 2023-11-15T05:58:09.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f441e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:09.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f441e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53664/0 (socket says 172.21.15.137:53664) 2023-11-15T05:58:09.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f441e59c700 1 -- 172.21.15.137:0/4181611726 learned_addr learned my addr 172.21.15.137:0/4181611726 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:09.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f441e59c700 1 -- 172.21.15.137:0/4181611726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4414005040 con 0x7f44200f5410 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.738+0000 7f441e59c700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4414013510 tx=0x7f441400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4414009320 con 0x7f44200f5410 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4414013f00 con 0x7f44200f5410 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4420060440 con 0x7f44200f5410 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4414014d90 con 0x7f44200f5410 2023-11-15T05:58:09.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4420109ae0 con 0x7f44200f5410 2023-11-15T05:58:09.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44140095f0 con 0x7f44200f5410 2023-11-15T05:58:09.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.739+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4420045ad0 con 0x7f44200f5410 2023-11-15T05:58:09.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.740+0000 7f440effd700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 0x7f4408032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:09.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.740+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4414040fd0 con 0x7f44200f5410 2023-11-15T05:58:09.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.740+0000 7f441dd9b700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 0x7f4408032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:09.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.740+0000 7f441dd9b700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 0x7f4408032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f44100096a0 tx=0x7f4410006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:09.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.745+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4414014420 con 0x7f44200f5410 2023-11-15T05:58:09.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.953+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44200fae60 con 0x7f44200f5410 2023-11-15T05:58:09.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.954+0000 7f440effd700 1 -- 172.21.15.137:0/4181611726 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f441401d7f0 con 0x7f44200f5410 2023-11-15T05:58:09.955 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:09.956 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:09.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 msgr2=0x7f4408032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:09.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 0x7f4408032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f44100096a0 tx=0x7f4410006b40).stop 2023-11-15T05:58:09.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 msgr2=0x7f4420109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:09.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4414013510 tx=0x7f441400b040).stop 2023-11-15T05:58:09.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 shutdown_connections 2023-11-15T05:58:09.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f440802ff60 0x7f4408032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:09.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 --2- 172.21.15.137:0/4181611726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44200f5410 0x7f4420109430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:09.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 >> 172.21.15.137:0/4181611726 conn(0x7f44200f0b70 msgr2=0x7f44200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:09.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 shutdown_connections 2023-11-15T05:58:09.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:09.955+0000 7f4424af1700 1 -- 172.21.15.137:0/4181611726 wait complete. 2023-11-15T05:58:09.964 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:08.996901+0000 mgr.smithi003.cldzke (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:09.955009+0000 mon.smithi003 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.137:0/4181611726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:11.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:11.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:10.997285+0000 mgr.smithi003.cldzke (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:12.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 -- 172.21.15.137:0/305316795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 msgr2=0x7fd2800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:12.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 --2- 172.21.15.137:0/305316795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fd274009fb0 tx=0x7fd274009b30).stop 2023-11-15T05:58:12.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 -- 172.21.15.137:0/305316795 shutdown_connections 2023-11-15T05:58:12.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 --2- 172.21.15.137:0/305316795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:12.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 -- 172.21.15.137:0/305316795 >> 172.21.15.137:0/305316795 conn(0x7fd2800f0b70 msgr2=0x7fd2800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:12.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 -- 172.21.15.137:0/305316795 shutdown_connections 2023-11-15T05:58:12.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.490+0000 7fd28512f700 1 -- 172.21.15.137:0/305316795 wait complete. 2023-11-15T05:58:12.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd28512f700 1 Processor -- start 2023-11-15T05:58:12.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd28512f700 1 -- start start 2023-11-15T05:58:12.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd28512f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:12.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd28512f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2800fb820 con 0x7fd2800f5410 2023-11-15T05:58:12.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd27ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd27ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53678/0 (socket says 172.21.15.137:53678) 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd27ed9d700 1 -- 172.21.15.137:0/1554559073 learned_addr learned my addr 172.21.15.137:0/1554559073 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd27ed9d700 1 -- 172.21.15.137:0/1554559073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd274005040 con 0x7fd2800f5410 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.491+0000 7fd27ed9d700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd274013510 tx=0x7fd27400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd274009320 con 0x7fd2800f5410 2023-11-15T05:58:12.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2800fb9c0 con 0x7fd2800f5410 2023-11-15T05:58:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd274013f00 con 0x7fd2800f5410 2023-11-15T05:58:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd274014d90 con 0x7fd2800f5410 2023-11-15T05:58:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2800fbe50 con 0x7fd2800f5410 2023-11-15T05:58:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2740095f0 con 0x7fd2800f5410 2023-11-15T05:58:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.492+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd280059100 con 0x7fd2800f5410 2023-11-15T05:58:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.493+0000 7fd2677fe700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 0x7fd268032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.493+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd274041000 con 0x7fd2800f5410 2023-11-15T05:58:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.493+0000 7fd27e59c700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 0x7fd268032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:12.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.493+0000 7fd27e59c700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 0x7fd268032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd2700096a0 tx=0x7fd270006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:12.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.498+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd274014420 con 0x7fd2800f5410 2023-11-15T05:58:12.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.706+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2800fae60 con 0x7fd2800f5410 2023-11-15T05:58:12.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.706+0000 7fd2677fe700 1 -- 172.21.15.137:0/1554559073 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd274010230 con 0x7fd2800f5410 2023-11-15T05:58:12.707 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:12.707 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 msgr2=0x7fd268032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 0x7fd268032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd2700096a0 tx=0x7fd270006b40).stop 2023-11-15T05:58:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 msgr2=0x7fd2800fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd274013510 tx=0x7fd27400b040).stop 2023-11-15T05:58:12.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 shutdown_connections 2023-11-15T05:58:12.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd26802ff60 0x7fd268032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:12.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 --2- 172.21.15.137:0/1554559073 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2800f5410 0x7fd2800fd3a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:12.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 >> 172.21.15.137:0/1554559073 conn(0x7fd2800f0b70 msgr2=0x7fd28005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:12.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 shutdown_connections 2023-11-15T05:58:12.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:12.708+0000 7fd28512f700 1 -- 172.21.15.137:0/1554559073 wait complete. 2023-11-15T05:58:12.711 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:13.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:12.706792+0000 mon.smithi003 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.137:0/1554559073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:14.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:14.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:14.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:12.997587+0000 mgr.smithi003.cldzke (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:15.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 -- 172.21.15.137:0/4189503727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 msgr2=0x7efca4069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:15.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 --2- 172.21.15.137:0/4189503727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca4069ec0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7efca0009fb0 tx=0x7efca0009b30).stop 2023-11-15T05:58:15.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 -- 172.21.15.137:0/4189503727 shutdown_connections 2023-11-15T05:58:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 --2- 172.21.15.137:0/4189503727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca4069ec0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 -- 172.21.15.137:0/4189503727 >> 172.21.15.137:0/4189503727 conn(0x7efca40f1c70 msgr2=0x7efca40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 -- 172.21.15.137:0/4189503727 shutdown_connections 2023-11-15T05:58:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.299+0000 7efcaba43700 1 -- 172.21.15.137:0/4189503727 wait complete. 2023-11-15T05:58:15.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efcaba43700 1 Processor -- start 2023-11-15T05:58:15.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efcaba43700 1 -- start start 2023-11-15T05:58:15.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efcaba43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:15.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efcaba43700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca4108de0 con 0x7efca406b9c0 2023-11-15T05:58:15.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efca97df700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:15.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efca97df700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53688/0 (socket says 172.21.15.137:53688) 2023-11-15T05:58:15.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.300+0000 7efca97df700 1 -- 172.21.15.137:0/2979509814 learned_addr learned my addr 172.21.15.137:0/2979509814 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:15.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efca97df700 1 -- 172.21.15.137:0/2979509814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0005040 con 0x7efca406b9c0 2023-11-15T05:58:15.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efca97df700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efca0013510 tx=0x7efca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:15.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efca0009320 con 0x7efca406b9c0 2023-11-15T05:58:15.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca4108fe0 con 0x7efca406b9c0 2023-11-15T05:58:15.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efca0013f00 con 0x7efca406b9c0 2023-11-15T05:58:15.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efca0014dc0 con 0x7efca406b9c0 2023-11-15T05:58:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.301+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca4109470 con 0x7efca406b9c0 2023-11-15T05:58:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.302+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efca0014420 con 0x7efca406b9c0 2023-11-15T05:58:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.302+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca4045ad0 con 0x7efca406b9c0 2023-11-15T05:58:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.302+0000 7efc99ffb700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.302+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efca00410b0 con 0x7efca406b9c0 2023-11-15T05:58:15.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.303+0000 7efca8fde700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:15.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.303+0000 7efca8fde700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 0x7efc90032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc940096a0 tx=0x7efc94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:15.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.307+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efca00095f0 con 0x7efca406b9c0 2023-11-15T05:58:15.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.510+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efca410a320 con 0x7efca406b9c0 2023-11-15T05:58:15.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.511+0000 7efc99ffb700 1 -- 172.21.15.137:0/2979509814 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7efca001cb10 con 0x7efca406b9c0 2023-11-15T05:58:15.512 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:15.512 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:15.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 msgr2=0x7efc90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:15.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 0x7efc90032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7efc940096a0 tx=0x7efc94006b40).stop 2023-11-15T05:58:15.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 msgr2=0x7efca41088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:15.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7efca0013510 tx=0x7efca000b040).stop 2023-11-15T05:58:15.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 shutdown_connections 2023-11-15T05:58:15.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efc9002ff60 0x7efc90032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:15.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 --2- 172.21.15.137:0/2979509814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efca406b9c0 0x7efca41088f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:15.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 >> 172.21.15.137:0/2979509814 conn(0x7efca40f1c70 msgr2=0x7efca40f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:15.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 shutdown_connections 2023-11-15T05:58:15.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:15.513+0000 7efcaba43700 1 -- 172.21.15.137:0/2979509814 wait complete. 2023-11-15T05:58:15.516 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:14.997838+0000 mgr.smithi003.cldzke (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:15.511757+0000 mon.smithi003 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.137:0/2979509814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:16.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:16.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:18.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- 172.21.15.137:0/3264868222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 msgr2=0x7fde8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:18.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 --2- 172.21.15.137:0/3264868222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c0f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fde7c009fb0 tx=0x7fde7c009b30).stop 2023-11-15T05:58:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- 172.21.15.137:0/3264868222 shutdown_connections 2023-11-15T05:58:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 --2- 172.21.15.137:0/3264868222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c0f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- 172.21.15.137:0/3264868222 >> 172.21.15.137:0/3264868222 conn(0x7fde8c0f0b70 msgr2=0x7fde8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- 172.21.15.137:0/3264868222 shutdown_connections 2023-11-15T05:58:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- 172.21.15.137:0/3264868222 wait complete. 2023-11-15T05:58:18.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 Processor -- start 2023-11-15T05:58:18.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- start start 2023-11-15T05:58:18.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:18.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.027+0000 7fde9141f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c105b20 con 0x7fde8c0f5410 2023-11-15T05:58:18.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde8affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:18.031 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde8affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:53694/0 (socket says 172.21.15.137:53694) 2023-11-15T05:58:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde8affd700 1 -- 172.21.15.137:0/3787715979 learned_addr learned my addr 172.21.15.137:0/3787715979 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde8affd700 1 -- 172.21.15.137:0/3787715979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c005040 con 0x7fde8c0f5410 2023-11-15T05:58:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde8affd700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fde7c013510 tx=0x7fde7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7c009320 con 0x7fde8c0f5410 2023-11-15T05:58:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c105d20 con 0x7fde8c0f5410 2023-11-15T05:58:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde7c013f00 con 0x7fde8c0f5410 2023-11-15T05:58:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7c014d90 con 0x7fde8c0f5410 2023-11-15T05:58:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.028+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c1061b0 con 0x7fde8c0f5410 2023-11-15T05:58:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.029+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde7c0263a0 con 0x7fde8c0f5410 2023-11-15T05:58:18.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.029+0000 7fde737fe700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 0x7fde74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.029+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde7c041090 con 0x7fde8c0f5410 2023-11-15T05:58:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.029+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c045ad0 con 0x7fde8c0f5410 2023-11-15T05:58:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.030+0000 7fde8a7fc700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 0x7fde74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.030+0000 7fde8a7fc700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 0x7fde74032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fde800096a0 tx=0x7fde80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:18.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.035+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde7c012920 con 0x7fde8c0f5410 2023-11-15T05:58:18.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.241+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde8c0fae60 con 0x7fde8c0f5410 2023-11-15T05:58:18.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.242+0000 7fde737fe700 1 -- 172.21.15.137:0/3787715979 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fde7c02b020 con 0x7fde8c0f5410 2023-11-15T05:58:18.243 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:18.243 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:18.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 msgr2=0x7fde74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:18.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 0x7fde74032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fde800096a0 tx=0x7fde80006b40).stop 2023-11-15T05:58:18.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 msgr2=0x7fde8c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:18.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fde7c013510 tx=0x7fde7c00b040).stop 2023-11-15T05:58:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 shutdown_connections 2023-11-15T05:58:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fde7402ff60 0x7fde74032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 --2- 172.21.15.137:0/3787715979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde8c0f5410 0x7fde8c105630 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 >> 172.21.15.137:0/3787715979 conn(0x7fde8c0f0b70 msgr2=0x7fde8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 shutdown_connections 2023-11-15T05:58:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:18.244+0000 7fde9141f700 1 -- 172.21.15.137:0/3787715979 wait complete. 2023-11-15T05:58:18.247 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:16.998061+0000 mgr.smithi003.cldzke (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:19.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:19 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:18.242421+0000 mon.smithi003 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.137:0/3787715979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:19.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:19.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:18.998383+0000 mgr.smithi003.cldzke (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:20.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- 172.21.15.137:0/1515875189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 msgr2=0x7f0d0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:20.787 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 --2- 172.21.15.137:0/1515875189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c0f90f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f0cfc009fb0 tx=0x7f0cfc009b30).stop 2023-11-15T05:58:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- 172.21.15.137:0/1515875189 shutdown_connections 2023-11-15T05:58:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 --2- 172.21.15.137:0/1515875189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c0f90f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- 172.21.15.137:0/1515875189 >> 172.21.15.137:0/1515875189 conn(0x7f0d0c0f0b70 msgr2=0x7f0d0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- 172.21.15.137:0/1515875189 shutdown_connections 2023-11-15T05:58:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- 172.21.15.137:0/1515875189 wait complete. 2023-11-15T05:58:20.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 Processor -- start 2023-11-15T05:58:20.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.785+0000 7f0d1475f700 1 -- start start 2023-11-15T05:58:20.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d1475f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:20.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d1475f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c106bd0 con 0x7f0d0c060280 2023-11-15T05:58:20.789 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d124fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:20.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d124fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44032/0 (socket says 172.21.15.137:44032) 2023-11-15T05:58:20.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d124fb700 1 -- 172.21.15.137:0/124919041 learned_addr learned my addr 172.21.15.137:0/124919041 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:20.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d124fb700 1 -- 172.21.15.137:0/124919041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc005040 con 0x7f0d0c060280 2023-11-15T05:58:20.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d124fb700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0cfc013510 tx=0x7f0cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:20.790 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cfc009320 con 0x7f0d0c060280 2023-11-15T05:58:20.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c106dd0 con 0x7f0d0c060280 2023-11-15T05:58:20.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cfc013f00 con 0x7f0d0c060280 2023-11-15T05:58:20.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cfc014dc0 con 0x7f0d0c060280 2023-11-15T05:58:20.791 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.786+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c107260 con 0x7f0d0c060280 2023-11-15T05:58:20.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.787+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cfc0095f0 con 0x7f0d0c060280 2023-11-15T05:58:20.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.787+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c045ad0 con 0x7f0d0c060280 2023-11-15T05:58:20.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.788+0000 7f0d02ffd700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 0x7f0cf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:20.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.788+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cfc041040 con 0x7f0d0c060280 2023-11-15T05:58:20.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.788+0000 7f0d11cfa700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 0x7f0cf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:20.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.788+0000 7f0d11cfa700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 0x7f0cf8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0d080096a0 tx=0x7f0d08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:20.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:20.793+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cfc014420 con 0x7f0d0c060280 2023-11-15T05:58:21.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.000+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d0c108110 con 0x7f0d0c060280 2023-11-15T05:58:21.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.000+0000 7f0d02ffd700 1 -- 172.21.15.137:0/124919041 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0cfc010230 con 0x7f0d0c060280 2023-11-15T05:58:21.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:21.001 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:21.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 msgr2=0x7f0cf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 0x7f0cf8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0d080096a0 tx=0x7f0d08006b40).stop 2023-11-15T05:58:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 msgr2=0x7f0d0c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0cfc013510 tx=0x7f0cfc00b040).stop 2023-11-15T05:58:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 shutdown_connections 2023-11-15T05:58:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cf802ff60 0x7f0cf8032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:21.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 --2- 172.21.15.137:0/124919041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d0c060280 0x7f0d0c1066e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:21.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 >> 172.21.15.137:0/124919041 conn(0x7f0d0c0f0b70 msgr2=0x7f0d0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:21.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 shutdown_connections 2023-11-15T05:58:21.004 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:21.002+0000 7f0d1475f700 1 -- 172.21.15.137:0/124919041 wait complete. 2023-11-15T05:58:21.005 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:20.998716+0000 mgr.smithi003.cldzke (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:21.000683+0000 mon.smithi003 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.137:0/124919041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:22.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:22.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:23.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.538+0000 7f15ef59e700 1 -- 172.21.15.137:0/4223848765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 msgr2=0x7f15f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.538+0000 7f15ef59e700 1 --2- 172.21.15.137:0/4223848765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f00f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f15e4009fb0 tx=0x7f15e4009b30).stop 2023-11-15T05:58:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.538+0000 7f15ef59e700 1 -- 172.21.15.137:0/4223848765 shutdown_connections 2023-11-15T05:58:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.538+0000 7f15ef59e700 1 --2- 172.21.15.137:0/4223848765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f00f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 -- 172.21.15.137:0/4223848765 >> 172.21.15.137:0/4223848765 conn(0x7f15f00f09a0 msgr2=0x7f15f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 -- 172.21.15.137:0/4223848765 shutdown_connections 2023-11-15T05:58:23.541 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 -- 172.21.15.137:0/4223848765 wait complete. 2023-11-15T05:58:23.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 Processor -- start 2023-11-15T05:58:23.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 -- start start 2023-11-15T05:58:23.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:23.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.539+0000 7f15ef59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f0060240 con 0x7f15f00f51a0 2023-11-15T05:58:23.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:23.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44048/0 (socket says 172.21.15.137:44048) 2023-11-15T05:58:23.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ee59c700 1 -- 172.21.15.137:0/1965834778 learned_addr learned my addr 172.21.15.137:0/1965834778 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:23.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ee59c700 1 -- 172.21.15.137:0/1965834778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4005040 con 0x7f15f00f51a0 2023-11-15T05:58:23.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ee59c700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f15e4013510 tx=0x7f15e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:23.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e4009320 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f0060440 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15e4013f00 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e4014dc0 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.540+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f0109860 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.541+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15e4014420 con 0x7f15f00f51a0 2023-11-15T05:58:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.541+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f0045ad0 con 0x7f15f00f51a0 2023-11-15T05:58:23.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.541+0000 7f15deffd700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:23.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.541+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15e4041110 con 0x7f15f00f51a0 2023-11-15T05:58:23.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.542+0000 7f15edd9b700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:23.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.542+0000 7f15edd9b700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 0x7f15d8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:23.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.546+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15e40095f0 con 0x7f15f00f51a0 2023-11-15T05:58:23.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.754+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15f010a080 con 0x7f15f00f51a0 2023-11-15T05:58:23.755 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.754+0000 7f15deffd700 1 -- 172.21.15.137:0/1965834778 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f15e401cb10 con 0x7f15f00f51a0 2023-11-15T05:58:23.756 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:23.756 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:23.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 msgr2=0x7f15d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:23.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 0x7f15d8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).stop 2023-11-15T05:58:23.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 msgr2=0x7f15f01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:23.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f15e4013510 tx=0x7f15e400b040).stop 2023-11-15T05:58:23.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 shutdown_connections 2023-11-15T05:58:23.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:23.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 --2- 172.21.15.137:0/1965834778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15f00f51a0 0x7f15f01091b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:23.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 >> 172.21.15.137:0/1965834778 conn(0x7f15f00f09a0 msgr2=0x7f15f00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:23.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 shutdown_connections 2023-11-15T05:58:23.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:23.756+0000 7f15ef59e700 1 -- 172.21.15.137:0/1965834778 wait complete. 2023-11-15T05:58:23.759 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:22.999045+0000 mgr.smithi003.cldzke (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:23.754966+0000 mon.smithi003 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.137:0/1965834778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:25.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:25.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:26.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.297+0000 7f1d481d7700 1 -- 172.21.15.137:0/611544774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 msgr2=0x7f1d400fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:26.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.297+0000 7f1d481d7700 1 --2- 172.21.15.137:0/611544774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d400fb550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f1d38009960 tx=0x7f1d38004d10).stop 2023-11-15T05:58:26.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.297+0000 7f1d481d7700 1 -- 172.21.15.137:0/611544774 shutdown_connections 2023-11-15T05:58:26.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.297+0000 7f1d481d7700 1 --2- 172.21.15.137:0/611544774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d400fb550 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:26.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.297+0000 7f1d481d7700 1 -- 172.21.15.137:0/611544774 >> 172.21.15.137:0/611544774 conn(0x7f1d400f0b70 msgr2=0x7f1d400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:26.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 -- 172.21.15.137:0/611544774 shutdown_connections 2023-11-15T05:58:26.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 -- 172.21.15.137:0/611544774 wait complete. 2023-11-15T05:58:26.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 Processor -- start 2023-11-15T05:58:26.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 -- start start 2023-11-15T05:58:26.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:26.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.298+0000 7f1d481d7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d40108da0 con 0x7f1d40060220 2023-11-15T05:58:26.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d45f73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:26.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d45f73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44080/0 (socket says 172.21.15.137:44080) 2023-11-15T05:58:26.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d45f73700 1 -- 172.21.15.137:0/2456205978 learned_addr learned my addr 172.21.15.137:0/2456205978 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:26.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d45f73700 1 -- 172.21.15.137:0/2456205978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d38009640 con 0x7f1d40060220 2023-11-15T05:58:26.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d45f73700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1d38007bd0 tx=0x7f1d38004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d38018070 con 0x7f1d40060220 2023-11-15T05:58:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d40108fa0 con 0x7f1d40060220 2023-11-15T05:58:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d3800b870 con 0x7f1d40060220 2023-11-15T05:58:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d3801d420 con 0x7f1d40060220 2023-11-15T05:58:26.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.299+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d40109430 con 0x7f1d40060220 2023-11-15T05:58:26.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.300+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d40045ad0 con 0x7f1d40060220 2023-11-15T05:58:26.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.300+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d38024060 con 0x7f1d40060220 2023-11-15T05:58:26.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.300+0000 7f1d367fc700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 0x7f1d2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:26.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.300+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d38041110 con 0x7f1d40060220 2023-11-15T05:58:26.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.301+0000 7f1d45772700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 0x7f1d2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:26.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.301+0000 7f1d45772700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 0x7f1d2c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1d300096a0 tx=0x7f1d30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:26.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.305+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d38011790 con 0x7f1d40060220 2023-11-15T05:58:26.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:24.999323+0000 mgr.smithi003.cldzke (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:26.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.515+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d4010a240 con 0x7f1d40060220 2023-11-15T05:58:26.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.515+0000 7f1d367fc700 1 -- 172.21.15.137:0/2456205978 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1d38011790 con 0x7f1d40060220 2023-11-15T05:58:26.516 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:26.517 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:26.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 msgr2=0x7f1d2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:26.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 0x7f1d2c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f1d300096a0 tx=0x7f1d30006b40).stop 2023-11-15T05:58:26.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 msgr2=0x7f1d401088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:26.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1d38007bd0 tx=0x7f1d38004d10).stop 2023-11-15T05:58:26.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 shutdown_connections 2023-11-15T05:58:26.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d2c02ff60 0x7f1d2c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:26.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 --2- 172.21.15.137:0/2456205978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d40060220 0x7f1d401088b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:26.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 >> 172.21.15.137:0/2456205978 conn(0x7f1d400f0b70 msgr2=0x7f1d400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:26.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 shutdown_connections 2023-11-15T05:58:26.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:26.517+0000 7f1d481d7700 1 -- 172.21.15.137:0/2456205978 wait complete. 2023-11-15T05:58:26.520 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:27.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:27 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:26.515937+0000 mon.smithi003 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.137:0/2456205978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:27.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:27.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:28.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:26.999656+0000 mgr.smithi003.cldzke (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:29.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 -- 172.21.15.137:0/2804534421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 msgr2=0x7f7d740f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:29.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/2804534421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d740f68a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7d68009fb0 tx=0x7f7d68009b30).stop 2023-11-15T05:58:29.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 -- 172.21.15.137:0/2804534421 shutdown_connections 2023-11-15T05:58:29.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/2804534421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d740f68a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:29.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 -- 172.21.15.137:0/2804534421 >> 172.21.15.137:0/2804534421 conn(0x7f7d740f1c00 msgr2=0x7f7d740f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:29.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 -- 172.21.15.137:0/2804534421 shutdown_connections 2023-11-15T05:58:29.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.113+0000 7f7d7c45f700 1 -- 172.21.15.137:0/2804534421 wait complete. 2023-11-15T05:58:29.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7c45f700 1 Processor -- start 2023-11-15T05:58:29.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7c45f700 1 -- start start 2023-11-15T05:58:29.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7c45f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:29.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7c45f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74106bc0 con 0x7f7d740f6480 2023-11-15T05:58:29.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7a1fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:29.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7a1fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:44104/0 (socket says 172.21.15.137:44104) 2023-11-15T05:58:29.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.114+0000 7f7d7a1fb700 1 -- 172.21.15.137:0/3692602641 learned_addr learned my addr 172.21.15.137:0/3692602641 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:29.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d7a1fb700 1 -- 172.21.15.137:0/3692602641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68005040 con 0x7f7d740f6480 2023-11-15T05:58:29.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d7a1fb700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7d68013510 tx=0x7f7d6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:29.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d68009320 con 0x7f7d740f6480 2023-11-15T05:58:29.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d74106dc0 con 0x7f7d740f6480 2023-11-15T05:58:29.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d68013f00 con 0x7f7d740f6480 2023-11-15T05:58:29.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d68014d90 con 0x7f7d740f6480 2023-11-15T05:58:29.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.115+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d74107250 con 0x7f7d740f6480 2023-11-15T05:58:29.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.116+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d680263a0 con 0x7f7d740f6480 2023-11-15T05:58:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.116+0000 7f7d66ffd700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 0x7f7d60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.116+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d74045ad0 con 0x7f7d740f6480 2023-11-15T05:58:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.116+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d680413c0 con 0x7f7d740f6480 2023-11-15T05:58:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.116+0000 7f7d799fa700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 0x7f7d60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.117+0000 7f7d799fa700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 0x7f7d60032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:29.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.121+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d68012920 con 0x7f7d740f6480 2023-11-15T05:58:29.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.324+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d74108100 con 0x7f7d740f6480 2023-11-15T05:58:29.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.325+0000 7f7d66ffd700 1 -- 172.21.15.137:0/3692602641 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7d6802b030 con 0x7f7d740f6480 2023-11-15T05:58:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:29.327 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:29.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.326+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 msgr2=0x7f7d60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:29.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.326+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 0x7f7d60032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).stop 2023-11-15T05:58:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.326+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 msgr2=0x7f7d741066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.326+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7d68013510 tx=0x7f7d6800b040).stop 2023-11-15T05:58:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 shutdown_connections 2023-11-15T05:58:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f7d6002ff60 0x7f7d60032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 --2- 172.21.15.137:0/3692602641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7d740f6480 0x7f7d741066d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:29.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 >> 172.21.15.137:0/3692602641 conn(0x7f7d740f1c00 msgr2=0x7f7d740f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:29.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 shutdown_connections 2023-11-15T05:58:29.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:29.327+0000 7f7d7c45f700 1 -- 172.21.15.137:0/3692602641 wait complete. 2023-11-15T05:58:29.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:30.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:28.999987+0000 mgr.smithi003.cldzke (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:29.326385+0000 mon.smithi003 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.137:0/3692602641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:30.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:30.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:31.000340+0000 mgr.smithi003.cldzke (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:31.948 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 -- 172.21.15.137:0/234352661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 msgr2=0x7f34500f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 --2- 172.21.15.137:0/234352661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34500f68a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f3440009fb0 tx=0x7f3440009b30).stop 2023-11-15T05:58:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 -- 172.21.15.137:0/234352661 shutdown_connections 2023-11-15T05:58:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 --2- 172.21.15.137:0/234352661 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34500f68a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 -- 172.21.15.137:0/234352661 >> 172.21.15.137:0/234352661 conn(0x7f34500f1c00 msgr2=0x7f34500f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.946+0000 7f345645b700 1 -- 172.21.15.137:0/234352661 shutdown_connections 2023-11-15T05:58:31.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.947+0000 7f345645b700 1 -- 172.21.15.137:0/234352661 wait complete. 2023-11-15T05:58:31.950 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.947+0000 7f345645b700 1 Processor -- start 2023-11-15T05:58:31.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.947+0000 7f345645b700 1 -- start start 2023-11-15T05:58:31.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.947+0000 7f345645b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:31.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.947+0000 7f345645b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450106bc0 con 0x7f34500f6480 2023-11-15T05:58:31.952 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:31.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57180/0 (socket says 172.21.15.137:57180) 2023-11-15T05:58:31.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344ffff700 1 -- 172.21.15.137:0/309542139 learned_addr learned my addr 172.21.15.137:0/309542139 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:31.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344ffff700 1 -- 172.21.15.137:0/309542139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440005040 con 0x7f34500f6480 2023-11-15T05:58:31.953 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344ffff700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3440013510 tx=0x7f344000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:31.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3440009320 con 0x7f34500f6480 2023-11-15T05:58:31.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3450106dc0 con 0x7f34500f6480 2023-11-15T05:58:31.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3440013f00 con 0x7f34500f6480 2023-11-15T05:58:31.954 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3440014d90 con 0x7f34500f6480 2023-11-15T05:58:31.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.948+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3450107250 con 0x7f34500f6480 2023-11-15T05:58:31.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.949+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3440014420 con 0x7f34500f6480 2023-11-15T05:58:31.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.949+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3450045ad0 con 0x7f34500f6480 2023-11-15T05:58:31.955 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.949+0000 7f344cff9700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:31.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.949+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3440041000 con 0x7f34500f6480 2023-11-15T05:58:31.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.950+0000 7f344f7fe700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:31.958 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.950+0000 7f344f7fe700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 0x7f343c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:31.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:31.954+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3440026bf0 con 0x7f34500f6480 2023-11-15T05:58:32.161 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.160+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3450108100 con 0x7f34500f6480 2023-11-15T05:58:32.162 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.161+0000 7f344cff9700 1 -- 172.21.15.137:0/309542139 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f344001db70 con 0x7f34500f6480 2023-11-15T05:58:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:32.163 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:32.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 msgr2=0x7f343c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:32.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 0x7f343c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).stop 2023-11-15T05:58:32.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 msgr2=0x7f34501066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:32.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3440013510 tx=0x7f344000b040).stop 2023-11-15T05:58:32.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 shutdown_connections 2023-11-15T05:58:32.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:32.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 --2- 172.21.15.137:0/309542139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f34500f6480 0x7f34501066d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:32.165 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 >> 172.21.15.137:0/309542139 conn(0x7f34500f1c00 msgr2=0x7f34500f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:32.166 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 shutdown_connections 2023-11-15T05:58:32.166 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:32.162+0000 7f345645b700 1 -- 172.21.15.137:0/309542139 wait complete. 2023-11-15T05:58:32.166 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:32.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:32.162256+0000 mon.smithi003 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.137:0/309542139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:33.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:33.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:33.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:33 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:33.000779+0000 mgr.smithi003.cldzke (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:34.742 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 -- 172.21.15.137:0/1256634261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 msgr2=0x7f81940f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:34.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 --2- 172.21.15.137:0/1256634261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f81940f4be0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8184009fb0 tx=0x7f8184009b30).stop 2023-11-15T05:58:34.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 -- 172.21.15.137:0/1256634261 shutdown_connections 2023-11-15T05:58:34.743 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 --2- 172.21.15.137:0/1256634261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f81940f4be0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:34.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 -- 172.21.15.137:0/1256634261 >> 172.21.15.137:0/1256634261 conn(0x7f81940f02e0 msgr2=0x7f81940f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:34.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 -- 172.21.15.137:0/1256634261 shutdown_connections 2023-11-15T05:58:34.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.740+0000 7f819938b700 1 -- 172.21.15.137:0/1256634261 wait complete. 2023-11-15T05:58:34.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.741+0000 7f819938b700 1 Processor -- start 2023-11-15T05:58:34.744 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.741+0000 7f819938b700 1 -- start start 2023-11-15T05:58:34.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.741+0000 7f819938b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:34.746 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.741+0000 7f819938b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8194106bf0 con 0x7f81940f47c0 2023-11-15T05:58:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f8192ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f8192ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57208/0 (socket says 172.21.15.137:57208) 2023-11-15T05:58:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f8192ffd700 1 -- 172.21.15.137:0/1854941786 learned_addr learned my addr 172.21.15.137:0/1854941786 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f8192ffd700 1 -- 172.21.15.137:0/1854941786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8184005040 con 0x7f81940f47c0 2023-11-15T05:58:34.747 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f8192ffd700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8184013510 tx=0x7f818400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:34.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8184009320 con 0x7f81940f47c0 2023-11-15T05:58:34.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8194106df0 con 0x7f81940f47c0 2023-11-15T05:58:34.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.742+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8184013f00 con 0x7f81940f47c0 2023-11-15T05:58:34.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.743+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8184014d90 con 0x7f81940f47c0 2023-11-15T05:58:34.748 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.743+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8194107280 con 0x7f81940f47c0 2023-11-15T05:58:34.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.743+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81840263a0 con 0x7f81940f47c0 2023-11-15T05:58:34.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.743+0000 7f817b7fe700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 0x7f817c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:34.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.743+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8184041300 con 0x7f81940f47c0 2023-11-15T05:58:34.749 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.744+0000 7f81927fc700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 0x7f817c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:34.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.744+0000 7f81927fc700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 0x7f817c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f81880096a0 tx=0x7f8188006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:34.750 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.744+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8194059100 con 0x7f81940f47c0 2023-11-15T05:58:34.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.749+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f818401d380 con 0x7f81940f47c0 2023-11-15T05:58:34.956 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.955+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8194045ad0 con 0x7f81940f47c0 2023-11-15T05:58:34.957 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.955+0000 7f817b7fe700 1 -- 172.21.15.137:0/1854941786 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f818402b030 con 0x7f81940f47c0 2023-11-15T05:58:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:34.958 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:34.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 msgr2=0x7f817c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:34.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 0x7f817c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f81880096a0 tx=0x7f8188006b40).stop 2023-11-15T05:58:34.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 msgr2=0x7f8194106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:34.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8184013510 tx=0x7f818400b040).stop 2023-11-15T05:58:34.959 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 shutdown_connections 2023-11-15T05:58:34.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f817c02ff60 0x7f817c032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:34.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 --2- 172.21.15.137:0/1854941786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81940f47c0 0x7f8194106700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:34.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 >> 172.21.15.137:0/1854941786 conn(0x7f81940f02e0 msgr2=0x7f81940f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:34.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 shutdown_connections 2023-11-15T05:58:34.960 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:34.957+0000 7f819938b700 1 -- 172.21.15.137:0/1854941786 wait complete. 2023-11-15T05:58:34.961 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:35.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:34.957036+0000 mon.smithi003 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.137:0/1854941786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:36.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:36.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:35.001121+0000 mgr.smithi003.cldzke (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:35.113385+0000 mon.smithi003 (mon.0) 199 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:58:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:35.116429+0000 mon.smithi003 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:58:37.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 -- 172.21.15.137:0/1459745445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 msgr2=0x7fb0240f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:37.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 --2- 172.21.15.137:0/1459745445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f9150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb014009fb0 tx=0x7fb014009b30).stop 2023-11-15T05:58:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 -- 172.21.15.137:0/1459745445 shutdown_connections 2023-11-15T05:58:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 --2- 172.21.15.137:0/1459745445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f9150 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 -- 172.21.15.137:0/1459745445 >> 172.21.15.137:0/1459745445 conn(0x7fb0240f0b70 msgr2=0x7fb0240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 -- 172.21.15.137:0/1459745445 shutdown_connections 2023-11-15T05:58:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.495+0000 7fb02b832700 1 -- 172.21.15.137:0/1459745445 wait complete. 2023-11-15T05:58:37.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb02b832700 1 Processor -- start 2023-11-15T05:58:37.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb02b832700 1 -- start start 2023-11-15T05:58:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb02b832700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb02b832700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0240f5050 con 0x7fb024060220 2023-11-15T05:58:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb0295ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb0295ce700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57230/0 (socket says 172.21.15.137:57230) 2023-11-15T05:58:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.496+0000 7fb0295ce700 1 -- 172.21.15.137:0/4252658895 learned_addr learned my addr 172.21.15.137:0/4252658895 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:37.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb0295ce700 1 -- 172.21.15.137:0/4252658895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014005040 con 0x7fb024060220 2023-11-15T05:58:37.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb0295ce700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb014013510 tx=0x7fb01400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:37.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb014009320 con 0x7fb024060220 2023-11-15T05:58:37.502 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0240f51f0 con 0x7fb024060220 2023-11-15T05:58:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb014013f00 con 0x7fb024060220 2023-11-15T05:58:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb014014dc0 con 0x7fb024060220 2023-11-15T05:58:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.497+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0240f5680 con 0x7fb024060220 2023-11-15T05:58:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.498+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb014014420 con 0x7fb024060220 2023-11-15T05:58:37.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.498+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb024059100 con 0x7fb024060220 2023-11-15T05:58:37.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.498+0000 7fb019ffb700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 0x7fb010032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:37.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.498+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb014041030 con 0x7fb024060220 2023-11-15T05:58:37.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.498+0000 7fb028dcd700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 0x7fb010032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:37.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.499+0000 7fb028dcd700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 0x7fb010032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fb0200096a0 tx=0x7fb020006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:37.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.503+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0140095f0 con 0x7fb024060220 2023-11-15T05:58:37.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.708+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0240fbf50 con 0x7fb024060220 2023-11-15T05:58:37.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.709+0000 7fb019ffb700 1 -- 172.21.15.137:0/4252658895 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb01401cb10 con 0x7fb024060220 2023-11-15T05:58:37.711 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:37.711 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:37.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 msgr2=0x7fb010032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 0x7fb010032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fb0200096a0 tx=0x7fb020006b40).stop 2023-11-15T05:58:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 msgr2=0x7fb0240f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb014013510 tx=0x7fb01400b040).stop 2023-11-15T05:58:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 shutdown_connections 2023-11-15T05:58:37.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb01002ff60 0x7fb010032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:37.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 --2- 172.21.15.137:0/4252658895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb024060220 0x7fb0240f6bd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:37.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 >> 172.21.15.137:0/4252658895 conn(0x7fb0240f0b70 msgr2=0x7fb0240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:37.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 shutdown_connections 2023-11-15T05:58:37.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:37.711+0000 7fb02b832700 1 -- 172.21.15.137:0/4252658895 wait complete. 2023-11-15T05:58:37.715 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:38.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:37.001490+0000 mgr.smithi003.cldzke (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:37.710573+0000 mon.smithi003 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.137:0/4252658895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:39.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:39.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:40.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 -- 172.21.15.137:0/577579846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 msgr2=0x7fee880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:40.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 --2- 172.21.15.137:0/577579846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee880f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fee7c009fb0 tx=0x7fee7c009b30).stop 2023-11-15T05:58:40.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 -- 172.21.15.137:0/577579846 shutdown_connections 2023-11-15T05:58:40.264 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 --2- 172.21.15.137:0/577579846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee880f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:40.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 -- 172.21.15.137:0/577579846 >> 172.21.15.137:0/577579846 conn(0x7fee880f1c70 msgr2=0x7fee880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:40.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 -- 172.21.15.137:0/577579846 shutdown_connections 2023-11-15T05:58:40.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.261+0000 7fee8e198700 1 -- 172.21.15.137:0/577579846 wait complete. 2023-11-15T05:58:40.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee8e198700 1 Processor -- start 2023-11-15T05:58:40.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee8e198700 1 -- start start 2023-11-15T05:58:40.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee8e198700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:40.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee8e198700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88106c20 con 0x7fee880f6510 2023-11-15T05:58:40.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:40.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55318/0 (socket says 172.21.15.137:55318) 2023-11-15T05:58:40.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.262+0000 7fee877fe700 1 -- 172.21.15.137:0/2831519750 learned_addr learned my addr 172.21.15.137:0/2831519750 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:40.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee877fe700 1 -- 172.21.15.137:0/2831519750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c005040 con 0x7fee880f6510 2023-11-15T05:58:40.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee877fe700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fee7c013510 tx=0x7fee7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:40.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee7c009320 con 0x7fee880f6510 2023-11-15T05:58:40.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee88106e20 con 0x7fee880f6510 2023-11-15T05:58:40.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee7c013f00 con 0x7fee880f6510 2023-11-15T05:58:40.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee7c014d90 con 0x7fee880f6510 2023-11-15T05:58:40.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.263+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee881072b0 con 0x7fee880f6510 2023-11-15T05:58:40.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.264+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee7c014420 con 0x7fee880f6510 2023-11-15T05:58:40.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.264+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee88045ad0 con 0x7fee880f6510 2023-11-15T05:58:40.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.264+0000 7fee6ffff700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 0x7fee70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:40.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.264+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee7c0410c0 con 0x7fee880f6510 2023-11-15T05:58:40.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.264+0000 7fee86ffd700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 0x7fee70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:40.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.265+0000 7fee86ffd700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 0x7fee70032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fee780096a0 tx=0x7fee78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:40.272 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.269+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee7c0095f0 con 0x7fee880f6510 2023-11-15T05:58:40.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:39.001806+0000 mgr.smithi003.cldzke (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:40.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.475+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee880fbf60 con 0x7fee880f6510 2023-11-15T05:58:40.477 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.476+0000 7fee6ffff700 1 -- 172.21.15.137:0/2831519750 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fee7c01db70 con 0x7fee880f6510 2023-11-15T05:58:40.477 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:40.478 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:40.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 msgr2=0x7fee70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:40.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 0x7fee70032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fee780096a0 tx=0x7fee78006b40).stop 2023-11-15T05:58:40.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 msgr2=0x7fee88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:40.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fee7c013510 tx=0x7fee7c00b040).stop 2023-11-15T05:58:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 shutdown_connections 2023-11-15T05:58:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fee7002ff60 0x7fee70032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 --2- 172.21.15.137:0/2831519750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee880f6510 0x7fee88106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 >> 172.21.15.137:0/2831519750 conn(0x7fee880f1c70 msgr2=0x7fee880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 shutdown_connections 2023-11-15T05:58:40.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:40.477+0000 7fee8e198700 1 -- 172.21.15.137:0/2831519750 wait complete. 2023-11-15T05:58:40.481 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:41.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:40.477015+0000 mon.smithi003 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.137:0/2831519750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:41.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:41.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:42.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:41.002154+0000 mgr.smithi003.cldzke (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:43.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.076+0000 7fe394a4f700 1 -- 172.21.15.137:0/2736344993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 msgr2=0x7fe3900f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:43.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.076+0000 7fe394a4f700 1 --2- 172.21.15.137:0/2736344993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3900f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fe384009fb0 tx=0x7fe384009b30).stop 2023-11-15T05:58:43.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.076+0000 7fe394a4f700 1 -- 172.21.15.137:0/2736344993 shutdown_connections 2023-11-15T05:58:43.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.076+0000 7fe394a4f700 1 --2- 172.21.15.137:0/2736344993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3900f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:43.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.076+0000 7fe394a4f700 1 -- 172.21.15.137:0/2736344993 >> 172.21.15.137:0/2736344993 conn(0x7fe3900f1aa0 msgr2=0x7fe3900f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:43.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.077+0000 7fe394a4f700 1 -- 172.21.15.137:0/2736344993 shutdown_connections 2023-11-15T05:58:43.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.077+0000 7fe394a4f700 1 -- 172.21.15.137:0/2736344993 wait complete. 2023-11-15T05:58:43.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.077+0000 7fe394a4f700 1 Processor -- start 2023-11-15T05:58:43.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.077+0000 7fe394a4f700 1 -- start start 2023-11-15T05:58:43.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe394a4f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:43.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe394a4f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3901069a0 con 0x7fe3900f62a0 2023-11-15T05:58:43.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe38ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:43.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe38ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55334/0 (socket says 172.21.15.137:55334) 2023-11-15T05:58:43.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe38ed9d700 1 -- 172.21.15.137:0/3352909811 learned_addr learned my addr 172.21.15.137:0/3352909811 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:43.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe38ed9d700 1 -- 172.21.15.137:0/3352909811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe384005040 con 0x7fe3900f62a0 2023-11-15T05:58:43.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe38ed9d700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe384013510 tx=0x7fe38400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:43.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.078+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe384009320 con 0x7fe3900f62a0 2023-11-15T05:58:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe390106ba0 con 0x7fe3900f62a0 2023-11-15T05:58:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe384013f00 con 0x7fe3900f62a0 2023-11-15T05:58:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe384014dc0 con 0x7fe3900f62a0 2023-11-15T05:58:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe390107030 con 0x7fe3900f62a0 2023-11-15T05:58:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3840095f0 con 0x7fe3900f62a0 2023-11-15T05:58:43.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe390045ad0 con 0x7fe3900f62a0 2023-11-15T05:58:43.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.079+0000 7fe3777fe700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 0x7fe378032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:43.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.080+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe384041040 con 0x7fe3900f62a0 2023-11-15T05:58:43.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.080+0000 7fe38e59c700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 0x7fe378032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:43.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.080+0000 7fe38e59c700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 0x7fe378032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe3800096a0 tx=0x7fe380006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:43.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.084+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe384014420 con 0x7fe3900f62a0 2023-11-15T05:58:43.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.289+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe390107ee0 con 0x7fe3900f62a0 2023-11-15T05:58:43.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.289+0000 7fe3777fe700 1 -- 172.21.15.137:0/3352909811 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe384010230 con 0x7fe3900f62a0 2023-11-15T05:58:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:43.292 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:43.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 msgr2=0x7fe378032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:43.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 0x7fe378032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe3800096a0 tx=0x7fe380006b40).stop 2023-11-15T05:58:43.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 msgr2=0x7fe3901064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:43.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe384013510 tx=0x7fe38400b040).stop 2023-11-15T05:58:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 shutdown_connections 2023-11-15T05:58:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe37802ff60 0x7fe378032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 --2- 172.21.15.137:0/3352909811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3900f62a0 0x7fe3901064b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 >> 172.21.15.137:0/3352909811 conn(0x7fe3900f1aa0 msgr2=0x7fe3900f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 shutdown_connections 2023-11-15T05:58:43.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:43.291+0000 7fe394a4f700 1 -- 172.21.15.137:0/3352909811 wait complete. 2023-11-15T05:58:43.295 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:43.002500+0000 mgr.smithi003.cldzke (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:43.291022+0000 mon.smithi003 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.137:0/3352909811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:44.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:44.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:45.827 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 -- 172.21.15.137:0/2408319005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 msgr2=0x7fc67c0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:45.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 --2- 172.21.15.137:0/2408319005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f9170 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc66c009fb0 tx=0x7fc66c009b30).stop 2023-11-15T05:58:45.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 -- 172.21.15.137:0/2408319005 shutdown_connections 2023-11-15T05:58:45.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 --2- 172.21.15.137:0/2408319005 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f9170 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:45.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 -- 172.21.15.137:0/2408319005 >> 172.21.15.137:0/2408319005 conn(0x7fc67c0f0b70 msgr2=0x7fc67c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:45.828 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 -- 172.21.15.137:0/2408319005 shutdown_connections 2023-11-15T05:58:45.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.825+0000 7fc681423700 1 -- 172.21.15.137:0/2408319005 wait complete. 2023-11-15T05:58:45.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc681423700 1 Processor -- start 2023-11-15T05:58:45.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc681423700 1 -- start start 2023-11-15T05:58:45.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc681423700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:45.829 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc681423700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67c0f5070 con 0x7fc67c060200 2023-11-15T05:58:45.831 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc67affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:45.831 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc67affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55356/0 (socket says 172.21.15.137:55356) 2023-11-15T05:58:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc67affd700 1 -- 172.21.15.137:0/3075646672 learned_addr learned my addr 172.21.15.137:0/3075646672 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.826+0000 7fc67affd700 1 -- 172.21.15.137:0/3075646672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c005040 con 0x7fc67c060200 2023-11-15T05:58:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc67affd700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc66c013510 tx=0x7fc66c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc66c009320 con 0x7fc67c060200 2023-11-15T05:58:45.832 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67c0f5210 con 0x7fc67c060200 2023-11-15T05:58:45.833 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67c0f56a0 con 0x7fc67c060200 2023-11-15T05:58:45.833 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc66c013f00 con 0x7fc67c060200 2023-11-15T05:58:45.833 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.827+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc66c014d90 con 0x7fc67c060200 2023-11-15T05:58:45.833 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.828+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc66c014420 con 0x7fc67c060200 2023-11-15T05:58:45.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.828+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67c059100 con 0x7fc67c060200 2023-11-15T05:58:45.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.828+0000 7fc6637fe700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 0x7fc664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:45.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.828+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc66c041310 con 0x7fc67c060200 2023-11-15T05:58:45.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.828+0000 7fc67a7fc700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 0x7fc664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:45.834 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.829+0000 7fc67a7fc700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 0x7fc664032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc6700096a0 tx=0x7fc670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:45.835 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:45.833+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc66c012b20 con 0x7fc67c060200 2023-11-15T05:58:46.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.038+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc67c0fbf50 con 0x7fc67c060200 2023-11-15T05:58:46.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.039+0000 7fc6637fe700 1 -- 172.21.15.137:0/3075646672 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc66c012b20 con 0x7fc67c060200 2023-11-15T05:58:46.041 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:46.041 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:46.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 msgr2=0x7fc664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:46.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 0x7fc664032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc6700096a0 tx=0x7fc670006b40).stop 2023-11-15T05:58:46.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 msgr2=0x7fc67c0f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fc66c013510 tx=0x7fc66c00b040).stop 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 shutdown_connections 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc66402ff60 0x7fc664032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 --2- 172.21.15.137:0/3075646672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc67c060200 0x7fc67c0f6bf0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 >> 172.21.15.137:0/3075646672 conn(0x7fc67c0f0b70 msgr2=0x7fc67c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 shutdown_connections 2023-11-15T05:58:46.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:46.040+0000 7fc681423700 1 -- 172.21.15.137:0/3075646672 wait complete. 2023-11-15T05:58:46.044 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:46.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:45.002819+0000 mgr.smithi003.cldzke (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:46.040102+0000 mon.smithi003 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.137:0/3075646672' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:47.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:47.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:48.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:47.003071+0000 mgr.smithi003.cldzke (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:48.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 -- 172.21.15.137:0/3104357940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 msgr2=0x7fe08c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:48.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 --2- 172.21.15.137:0/3104357940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c0f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fe07c009fb0 tx=0x7fe07c009b30).stop 2023-11-15T05:58:48.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 -- 172.21.15.137:0/3104357940 shutdown_connections 2023-11-15T05:58:48.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 --2- 172.21.15.137:0/3104357940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c0f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:48.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 -- 172.21.15.137:0/3104357940 >> 172.21.15.137:0/3104357940 conn(0x7fe08c0f09a0 msgr2=0x7fe08c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:48.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 -- 172.21.15.137:0/3104357940 shutdown_connections 2023-11-15T05:58:48.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 -- 172.21.15.137:0/3104357940 wait complete. 2023-11-15T05:58:48.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.518+0000 7fe09169f700 1 Processor -- start 2023-11-15T05:58:48.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe09169f700 1 -- start start 2023-11-15T05:58:48.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe09169f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:48.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe09169f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1058a0 con 0x7fe08c0f51a0 2023-11-15T05:58:48.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe08bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:48.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe08bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55368/0 (socket says 172.21.15.137:55368) 2023-11-15T05:58:48.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe08bfff700 1 -- 172.21.15.137:0/1946385565 learned_addr learned my addr 172.21.15.137:0/1946385565 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:48.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe08bfff700 1 -- 172.21.15.137:0/1946385565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c005040 con 0x7fe08c0f51a0 2023-11-15T05:58:48.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.519+0000 7fe08bfff700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe07c009830 tx=0x7fe07c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:48.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.520+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07c009320 con 0x7fe08c0f51a0 2023-11-15T05:58:48.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.520+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c105aa0 con 0x7fe08c0f51a0 2023-11-15T05:58:48.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.520+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe07c0136c0 con 0x7fe08c0f51a0 2023-11-15T05:58:48.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.520+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe07c01ee90 con 0x7fe08c0f51a0 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.520+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c105f30 con 0x7fe08c0f51a0 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.521+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe07c01e420 con 0x7fe08c0f51a0 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.521+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c045ad0 con 0x7fe08c0f51a0 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.521+0000 7fe088ff9700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 0x7fe0740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.521+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe07c040f10 con 0x7fe08c0f51a0 2023-11-15T05:58:48.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.521+0000 7fe08b7fe700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 0x7fe0740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:48.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.522+0000 7fe08b7fe700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 0x7fe0740323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:48.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.526+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe07c01c790 con 0x7fe08c0f51a0 2023-11-15T05:58:48.734 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.732+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe08c106d80 con 0x7fe08c0f51a0 2023-11-15T05:58:48.735 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.733+0000 7fe088ff9700 1 -- 172.21.15.137:0/1946385565 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe07c02c2c0 con 0x7fe08c0f51a0 2023-11-15T05:58:48.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:48.736 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:48.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 msgr2=0x7fe0740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:48.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 0x7fe0740323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).stop 2023-11-15T05:58:48.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 msgr2=0x7fe08c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:48.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fe07c009830 tx=0x7fe07c00b040).stop 2023-11-15T05:58:48.737 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 shutdown_connections 2023-11-15T05:58:48.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07402ff10 0x7fe0740323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:48.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 --2- 172.21.15.137:0/1946385565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe08c0f51a0 0x7fe08c1053b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:48.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 >> 172.21.15.137:0/1946385565 conn(0x7fe08c0f09a0 msgr2=0x7fe08c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:48.738 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 shutdown_connections 2023-11-15T05:58:48.739 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:48.735+0000 7fe09169f700 1 -- 172.21.15.137:0/1946385565 wait complete. 2023-11-15T05:58:48.739 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:49.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:49 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:48.734670+0000 mon.smithi003 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.137:0/1946385565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:50.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:50.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:49.003382+0000 mgr.smithi003.cldzke (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:51.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 -- 172.21.15.137:0/1239722734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 msgr2=0x7fafa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:51.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 --2- 172.21.15.137:0/1239722734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa00f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7faf94009fb0 tx=0x7faf94009b30).stop 2023-11-15T05:58:51.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 -- 172.21.15.137:0/1239722734 shutdown_connections 2023-11-15T05:58:51.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 --2- 172.21.15.137:0/1239722734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa00f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:51.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 -- 172.21.15.137:0/1239722734 >> 172.21.15.137:0/1239722734 conn(0x7fafa00f0b70 msgr2=0x7fafa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:51.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 -- 172.21.15.137:0/1239722734 shutdown_connections 2023-11-15T05:58:51.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.291+0000 7fafa6f21700 1 -- 172.21.15.137:0/1239722734 wait complete. 2023-11-15T05:58:51.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa6f21700 1 Processor -- start 2023-11-15T05:58:51.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa6f21700 1 -- start start 2023-11-15T05:58:51.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa6f21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:51.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa6f21700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0060240 con 0x7fafa00f5410 2023-11-15T05:58:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa4cbd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa4cbd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46450/0 (socket says 172.21.15.137:46450) 2023-11-15T05:58:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.292+0000 7fafa4cbd700 1 -- 172.21.15.137:0/3269538789 learned_addr learned my addr 172.21.15.137:0/3269538789 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7fafa4cbd700 1 -- 172.21.15.137:0/3269538789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94005040 con 0x7fafa00f5410 2023-11-15T05:58:51.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7fafa4cbd700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7faf94013510 tx=0x7faf9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:51.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf94009320 con 0x7fafa00f5410 2023-11-15T05:58:51.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa0060440 con 0x7fafa00f5410 2023-11-15T05:58:51.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa0109a80 con 0x7fafa00f5410 2023-11-15T05:58:51.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf94013f00 con 0x7fafa00f5410 2023-11-15T05:58:51.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.293+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf94014d90 con 0x7fafa00f5410 2023-11-15T05:58:51.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.294+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf94014420 con 0x7fafa00f5410 2023-11-15T05:58:51.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.294+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0045ad0 con 0x7fafa00f5410 2023-11-15T05:58:51.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.294+0000 7faf9d7fa700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:51.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.294+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf94041090 con 0x7fafa00f5410 2023-11-15T05:58:51.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.294+0000 7faf9ffff700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:51.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.295+0000 7faf9ffff700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 0x7faf90032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf8c0096a0 tx=0x7faf8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:51.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.299+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf9406d2b0 con 0x7fafa00f5410 2023-11-15T05:58:51.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.505+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafa010a290 con 0x7fafa00f5410 2023-11-15T05:58:51.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.506+0000 7faf9d7fa700 1 -- 172.21.15.137:0/3269538789 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7faf940095f0 con 0x7fafa00f5410 2023-11-15T05:58:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:51.508 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:51.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 msgr2=0x7faf90032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:51.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 0x7faf90032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf8c0096a0 tx=0x7faf8c006b40).stop 2023-11-15T05:58:51.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 msgr2=0x7fafa01093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:51.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7faf94013510 tx=0x7faf9400b040).stop 2023-11-15T05:58:51.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 shutdown_connections 2023-11-15T05:58:51.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faf9002ffb0 0x7faf90032470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:51.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 --2- 172.21.15.137:0/3269538789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fafa00f5410 0x7fafa01093d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:51.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 >> 172.21.15.137:0/3269538789 conn(0x7fafa00f0b70 msgr2=0x7fafa00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:51.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 shutdown_connections 2023-11-15T05:58:51.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:51.507+0000 7fafa6f21700 1 -- 172.21.15.137:0/3269538789 wait complete. 2023-11-15T05:58:51.511 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:51.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:51.003731+0000 mgr.smithi003.cldzke (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:51.507109+0000 mon.smithi003 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.137:0/3269538789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:52.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:52.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:54.032 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 -- 172.21.15.137:0/1466773321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 msgr2=0x7f0b700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:54.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 --2- 172.21.15.137:0/1466773321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b700f6930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0b58009fb0 tx=0x7f0b58009b30).stop 2023-11-15T05:58:54.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 -- 172.21.15.137:0/1466773321 shutdown_connections 2023-11-15T05:58:54.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 --2- 172.21.15.137:0/1466773321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b700f6930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:54.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 -- 172.21.15.137:0/1466773321 >> 172.21.15.137:0/1466773321 conn(0x7f0b700f1c70 msgr2=0x7f0b700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:54.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 -- 172.21.15.137:0/1466773321 shutdown_connections 2023-11-15T05:58:54.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.029+0000 7f0b75be3700 1 -- 172.21.15.137:0/1466773321 wait complete. 2023-11-15T05:58:54.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.030+0000 7f0b75be3700 1 Processor -- start 2023-11-15T05:58:54.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.030+0000 7f0b75be3700 1 -- start start 2023-11-15T05:58:54.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.030+0000 7f0b75be3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:54.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.030+0000 7f0b75be3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b70106c20 con 0x7f0b700f6510 2023-11-15T05:58:54.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:54.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46472/0 (socket says 172.21.15.137:46472) 2023-11-15T05:58:54.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b6f7fe700 1 -- 172.21.15.137:0/3295244788 learned_addr learned my addr 172.21.15.137:0/3295244788 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:54.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b6f7fe700 1 -- 172.21.15.137:0/3295244788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58005040 con 0x7f0b700f6510 2023-11-15T05:58:54.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b6f7fe700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0b58013510 tx=0x7f0b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:54.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.031+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b58009320 con 0x7f0b700f6510 2023-11-15T05:58:54.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b58013f00 con 0x7f0b700f6510 2023-11-15T05:58:54.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b70106e20 con 0x7f0b700f6510 2023-11-15T05:58:54.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b58014d90 con 0x7f0b700f6510 2023-11-15T05:58:54.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b701072b0 con 0x7f0b700f6510 2023-11-15T05:58:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b580095f0 con 0x7f0b700f6510 2023-11-15T05:58:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.032+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b70045ad0 con 0x7f0b700f6510 2023-11-15T05:58:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.033+0000 7f0b67fff700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 0x7f0b5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.033+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b580410a0 con 0x7f0b700f6510 2023-11-15T05:58:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.033+0000 7f0b6effd700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 0x7f0b5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:54.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.033+0000 7f0b6effd700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 0x7f0b5c032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0b600096a0 tx=0x7f0b60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:54.040 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.037+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b5801c320 con 0x7f0b700f6510 2023-11-15T05:58:54.242 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.241+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b700fc050 con 0x7f0b700f6510 2023-11-15T05:58:54.243 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.242+0000 7f0b67fff700 1 -- 172.21.15.137:0/3295244788 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0b58010230 con 0x7f0b700f6510 2023-11-15T05:58:54.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:54.244 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:54.245 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 msgr2=0x7f0b5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:54.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 0x7f0b5c032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0b600096a0 tx=0x7f0b60006b40).stop 2023-11-15T05:58:54.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 msgr2=0x7f0b70106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:54.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0b58013510 tx=0x7f0b5800b040).stop 2023-11-15T05:58:54.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 shutdown_connections 2023-11-15T05:58:54.246 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0b5c02ffb0 0x7f0b5c032470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:54.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 --2- 172.21.15.137:0/3295244788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b700f6510 0x7f0b70106730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:54.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 >> 172.21.15.137:0/3295244788 conn(0x7f0b700f1c70 msgr2=0x7f0b700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:54.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 shutdown_connections 2023-11-15T05:58:54.247 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:54.243+0000 7f0b75be3700 1 -- 172.21.15.137:0/3295244788 wait complete. 2023-11-15T05:58:54.247 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:54.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:53.004021+0000 mgr.smithi003.cldzke (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:55.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:55 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:54.243173+0000 mon.smithi003 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.137:0/3295244788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:55.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:55.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:56.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:55.004263+0000 mgr.smithi003.cldzke (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:56.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 -- 172.21.15.137:0/1218573398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 msgr2=0x7f0c480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/1218573398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c480f6930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f0c44009fb0 tx=0x7f0c44009b30).stop 2023-11-15T05:58:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 -- 172.21.15.137:0/1218573398 shutdown_connections 2023-11-15T05:58:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/1218573398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c480f6930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 -- 172.21.15.137:0/1218573398 >> 172.21.15.137:0/1218573398 conn(0x7f0c480f1c70 msgr2=0x7f0c480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 -- 172.21.15.137:0/1218573398 shutdown_connections 2023-11-15T05:58:56.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.766+0000 7f0c4f96c700 1 -- 172.21.15.137:0/1218573398 wait complete. 2023-11-15T05:58:56.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4f96c700 1 Processor -- start 2023-11-15T05:58:56.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4f96c700 1 -- start start 2023-11-15T05:58:56.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4f96c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:56.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4f96c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48106c20 con 0x7f0c480f6510 2023-11-15T05:58:56.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4d708700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4d708700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46482/0 (socket says 172.21.15.137:46482) 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4d708700 1 -- 172.21.15.137:0/983618894 learned_addr learned my addr 172.21.15.137:0/983618894 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.767+0000 7f0c4d708700 1 -- 172.21.15.137:0/983618894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44005040 con 0x7f0c480f6510 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c4d708700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f0c44014000 tx=0x7f0c4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c440095f0 con 0x7f0c480f6510 2023-11-15T05:58:56.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c44007e70 con 0x7f0c480f6510 2023-11-15T05:58:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c44014ea0 con 0x7f0c480f6510 2023-11-15T05:58:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c48106e20 con 0x7f0c480f6510 2023-11-15T05:58:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.768+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c481072b0 con 0x7f0c480f6510 2023-11-15T05:58:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.769+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c44025070 con 0x7f0c480f6510 2023-11-15T05:58:56.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.769+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c48045ad0 con 0x7f0c480f6510 2023-11-15T05:58:56.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.769+0000 7f0c3dffb700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 0x7f0c34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:56.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.769+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c44040a40 con 0x7f0c480f6510 2023-11-15T05:58:56.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.769+0000 7f0c4cf07700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 0x7f0c34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:56.775 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.770+0000 7f0c4cf07700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 0x7f0c34032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0c380096a0 tx=0x7f0c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:56.776 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.774+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c440079c0 con 0x7f0c480f6510 2023-11-15T05:58:56.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.983+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c480fbf60 con 0x7f0c480f6510 2023-11-15T05:58:56.985 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.985+0000 7f0c3dffb700 1 -- 172.21.15.137:0/983618894 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0c4400f300 con 0x7f0c480f6510 2023-11-15T05:58:56.986 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:56.986 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:56.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 msgr2=0x7f0c34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:56.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 0x7f0c34032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0c380096a0 tx=0x7f0c38006b40).stop 2023-11-15T05:58:56.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 msgr2=0x7f0c48106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:56.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f0c44014000 tx=0x7f0c4400b040).stop 2023-11-15T05:58:56.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 shutdown_connections 2023-11-15T05:58:56.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0c3402ff60 0x7f0c34032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:56.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 --2- 172.21.15.137:0/983618894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c480f6510 0x7f0c48106730 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:56.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 >> 172.21.15.137:0/983618894 conn(0x7f0c480f1c70 msgr2=0x7f0c480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:56.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 shutdown_connections 2023-11-15T05:58:56.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:56.986+0000 7f0c4f96c700 1 -- 172.21.15.137:0/983618894 wait complete. 2023-11-15T05:58:56.990 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:58:57.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:57 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:56.985480+0000 mon.smithi003 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.137:0/983618894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:58:58.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:58:58.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:58:58.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:58:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:57.004566+0000 mgr.smithi003.cldzke (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:58:59.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 -- 172.21.15.137:0/4063238415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 msgr2=0x7f9ec40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:59.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 --2- 172.21.15.137:0/4063238415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec40f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9eb8009fb0 tx=0x7f9eb8009b30).stop 2023-11-15T05:58:59.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 -- 172.21.15.137:0/4063238415 shutdown_connections 2023-11-15T05:58:59.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 --2- 172.21.15.137:0/4063238415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec40f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:59.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 -- 172.21.15.137:0/4063238415 >> 172.21.15.137:0/4063238415 conn(0x7f9ec40f0b70 msgr2=0x7f9ec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:59.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 -- 172.21.15.137:0/4063238415 shutdown_connections 2023-11-15T05:58:59.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.557+0000 7f9ec984e700 1 -- 172.21.15.137:0/4063238415 wait complete. 2023-11-15T05:58:59.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec984e700 1 Processor -- start 2023-11-15T05:58:59.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec984e700 1 -- start start 2023-11-15T05:58:59.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec984e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:59.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec984e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec4105b20 con 0x7f9ec40f5410 2023-11-15T05:58:59.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:59.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57384/0 (socket says 172.21.15.137:57384) 2023-11-15T05:58:59.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.558+0000 7f9ec2ffd700 1 -- 172.21.15.137:0/403038014 learned_addr learned my addr 172.21.15.137:0/403038014 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9ec2ffd700 1 -- 172.21.15.137:0/403038014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8005040 con 0x7f9ec40f5410 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9ec2ffd700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9eb8014000 tx=0x7f9eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb80095f0 con 0x7f9ec40f5410 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec4105d20 con 0x7f9ec40f5410 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9eb8007e70 con 0x7f9ec40f5410 2023-11-15T05:58:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9eb8014ea0 con 0x7f9ec40f5410 2023-11-15T05:58:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.559+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec41061b0 con 0x7f9ec40f5410 2023-11-15T05:58:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.560+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9eb8025070 con 0x7f9ec40f5410 2023-11-15T05:58:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.560+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec4045ad0 con 0x7f9ec40f5410 2023-11-15T05:58:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.560+0000 7f9eab7fe700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:58:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.560+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9eb8040a30 con 0x7f9ec40f5410 2023-11-15T05:58:59.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.560+0000 7f9ec27fc700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:58:59.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.561+0000 7f9ec27fc700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 0x7f9eac032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:58:59.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.565+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9eb80079c0 con 0x7f9ec40f5410 2023-11-15T05:58:59.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.771+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec40fae60 con 0x7f9ec40f5410 2023-11-15T05:58:59.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.772+0000 7f9eab7fe700 1 -- 172.21.15.137:0/403038014 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9eb800f300 con 0x7f9ec40f5410 2023-11-15T05:58:59.775 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:58:59.775 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:58:59.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 msgr2=0x7f9eac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:59.777 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 0x7f9eac032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).stop 2023-11-15T05:58:59.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 msgr2=0x7f9ec4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:58:59.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9eb8014000 tx=0x7f9eb800b040).stop 2023-11-15T05:58:59.778 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 shutdown_connections 2023-11-15T05:58:59.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:59.779 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 --2- 172.21.15.137:0/403038014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ec40f5410 0x7f9ec4105630 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:58:59.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 >> 172.21.15.137:0/403038014 conn(0x7f9ec40f0b70 msgr2=0x7f9ec40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:58:59.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 shutdown_connections 2023-11-15T05:58:59.780 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:58:59.774+0000 7f9ec984e700 1 -- 172.21.15.137:0/403038014 wait complete. 2023-11-15T05:58:59.781 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:58:59.004896+0000 mgr.smithi003.cldzke (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:58:59.773421+0000 mon.smithi003 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.137:0/403038014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:01.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:01.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:01.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:01.005272+0000 mgr.smithi003.cldzke (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:02.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 -- 172.21.15.137:0/4105754590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 msgr2=0x7f0fb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:02.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 --2- 172.21.15.137:0/4105754590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb80f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0fa8009fb0 tx=0x7f0fa8009b30).stop 2023-11-15T05:59:02.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 -- 172.21.15.137:0/4105754590 shutdown_connections 2023-11-15T05:59:02.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 --2- 172.21.15.137:0/4105754590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb80f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:02.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 -- 172.21.15.137:0/4105754590 >> 172.21.15.137:0/4105754590 conn(0x7f0fb80f1c70 msgr2=0x7f0fb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:02.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 -- 172.21.15.137:0/4105754590 shutdown_connections 2023-11-15T05:59:02.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.315+0000 7f0fc0675700 1 -- 172.21.15.137:0/4105754590 wait complete. 2023-11-15T05:59:02.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.316+0000 7f0fc0675700 1 Processor -- start 2023-11-15T05:59:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.316+0000 7f0fc0675700 1 -- start start 2023-11-15T05:59:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.316+0000 7f0fc0675700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.316+0000 7f0fc0675700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8106c20 con 0x7f0fb80f6510 2023-11-15T05:59:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fbe411700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:02.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fbe411700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57412/0 (socket says 172.21.15.137:57412) 2023-11-15T05:59:02.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fbe411700 1 -- 172.21.15.137:0/1735736440 learned_addr learned my addr 172.21.15.137:0/1735736440 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:02.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fbe411700 1 -- 172.21.15.137:0/1735736440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa8005040 con 0x7f0fb80f6510 2023-11-15T05:59:02.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fbe411700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0fa8013510 tx=0x7f0fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:02.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa8009320 con 0x7f0fb80f6510 2023-11-15T05:59:02.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb8106e20 con 0x7f0fb80f6510 2023-11-15T05:59:02.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fa8013f00 con 0x7f0fb80f6510 2023-11-15T05:59:02.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fa8014d90 con 0x7f0fb80f6510 2023-11-15T05:59:02.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.317+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb81072b0 con 0x7f0fb80f6510 2023-11-15T05:59:02.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.318+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fa80095f0 con 0x7f0fb80f6510 2023-11-15T05:59:02.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.318+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb8045ad0 con 0x7f0fb80f6510 2023-11-15T05:59:02.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.318+0000 7f0faeffd700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 0x7f0fa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:02.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.318+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fa8041000 con 0x7f0fb80f6510 2023-11-15T05:59:02.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.318+0000 7f0fbdc10700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 0x7f0fa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:02.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.319+0000 7f0fbdc10700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 0x7f0fa4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0fb40096a0 tx=0x7f0fb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:02.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.323+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fa8014420 con 0x7f0fb80f6510 2023-11-15T05:59:02.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.530+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fb80fbf60 con 0x7f0fb80f6510 2023-11-15T05:59:02.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.531+0000 7f0faeffd700 1 -- 172.21.15.137:0/1735736440 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0fa801d7f0 con 0x7f0fb80f6510 2023-11-15T05:59:02.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:02.533 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:02.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 msgr2=0x7f0fa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:02.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 0x7f0fa4032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0fb40096a0 tx=0x7f0fb4006b40).stop 2023-11-15T05:59:02.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 msgr2=0x7f0fb8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:02.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0fa8013510 tx=0x7f0fa800b040).stop 2023-11-15T05:59:02.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 shutdown_connections 2023-11-15T05:59:02.538 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0fa402ff60 0x7f0fa4032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 --2- 172.21.15.137:0/1735736440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fb80f6510 0x7f0fb8106730 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 >> 172.21.15.137:0/1735736440 conn(0x7f0fb80f1c70 msgr2=0x7f0fb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 shutdown_connections 2023-11-15T05:59:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:02.532+0000 7f0fc0675700 1 -- 172.21.15.137:0/1735736440 wait complete. 2023-11-15T05:59:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:02.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:02.532131+0000 mon.smithi003 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.137:0/1735736440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:03.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:03.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:03.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:03.005537+0000 mgr.smithi003.cldzke (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:05.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.089+0000 7f4986f6e700 1 -- 172.21.15.137:0/3519248664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 msgr2=0x7f49800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:05.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.089+0000 7f4986f6e700 1 --2- 172.21.15.137:0/3519248664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f49800f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f4974009fb0 tx=0x7f4974009b30).stop 2023-11-15T05:59:05.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- 172.21.15.137:0/3519248664 shutdown_connections 2023-11-15T05:59:05.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 --2- 172.21.15.137:0/3519248664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f49800f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:05.092 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- 172.21.15.137:0/3519248664 >> 172.21.15.137:0/3519248664 conn(0x7f49800f0b70 msgr2=0x7f49800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:05.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- 172.21.15.137:0/3519248664 shutdown_connections 2023-11-15T05:59:05.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- 172.21.15.137:0/3519248664 wait complete. 2023-11-15T05:59:05.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 Processor -- start 2023-11-15T05:59:05.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- start start 2023-11-15T05:59:05.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:05.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.090+0000 7f4986f6e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4980060240 con 0x7f49800f5410 2023-11-15T05:59:05.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4984d0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:05.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4984d0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57418/0 (socket says 172.21.15.137:57418) 2023-11-15T05:59:05.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4984d0a700 1 -- 172.21.15.137:0/1624306576 learned_addr learned my addr 172.21.15.137:0/1624306576 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:05.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4984d0a700 1 -- 172.21.15.137:0/1624306576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974005040 con 0x7f49800f5410 2023-11-15T05:59:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4984d0a700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4974014000 tx=0x7f497400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49740095f0 con 0x7f49800f5410 2023-11-15T05:59:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4980060440 con 0x7f49800f5410 2023-11-15T05:59:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4974007e70 con 0x7f49800f5410 2023-11-15T05:59:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4974014ea0 con 0x7f49800f5410 2023-11-15T05:59:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.091+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4980109ae0 con 0x7f49800f5410 2023-11-15T05:59:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.092+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4974025070 con 0x7f49800f5410 2023-11-15T05:59:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.092+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4980045ad0 con 0x7f49800f5410 2023-11-15T05:59:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.092+0000 7f497d7fa700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 0x7f4970032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.092+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49740409f0 con 0x7f49800f5410 2023-11-15T05:59:05.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.093+0000 7f497ffff700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 0x7f4970032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:05.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.093+0000 7f497ffff700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 0x7f4970032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f496c0096a0 tx=0x7f496c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:05.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.098+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49740079c0 con 0x7f49800f5410 2023-11-15T05:59:05.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.303+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49800fae60 con 0x7f49800f5410 2023-11-15T05:59:05.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f497d7fa700 1 -- 172.21.15.137:0/1624306576 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f497400f300 con 0x7f49800f5410 2023-11-15T05:59:05.306 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:05.306 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:05.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 msgr2=0x7f4970032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:05.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 0x7f4970032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f496c0096a0 tx=0x7f496c006b40).stop 2023-11-15T05:59:05.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 msgr2=0x7f4980109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:05.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f4974014000 tx=0x7f497400b040).stop 2023-11-15T05:59:05.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 shutdown_connections 2023-11-15T05:59:05.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f497002ff60 0x7f4970032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:05.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 --2- 172.21.15.137:0/1624306576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49800f5410 0x7f4980109430 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:05.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 >> 172.21.15.137:0/1624306576 conn(0x7f49800f0b70 msgr2=0x7f49800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:05.308 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 shutdown_connections 2023-11-15T05:59:05.309 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:05.305+0000 7f4986f6e700 1 -- 172.21.15.137:0/1624306576 wait complete. 2023-11-15T05:59:05.309 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:05.005770+0000 mgr.smithi003.cldzke (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:05.305220+0000 mon.smithi003 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.137:0/1624306576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:06.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:06.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:07.861 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 -- 172.21.15.137:0/31975649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 msgr2=0x7faa780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 --2- 172.21.15.137:0/31975649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa780f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7faa68009fb0 tx=0x7faa68009b30).stop 2023-11-15T05:59:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 -- 172.21.15.137:0/31975649 shutdown_connections 2023-11-15T05:59:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 --2- 172.21.15.137:0/31975649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa780f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 -- 172.21.15.137:0/31975649 >> 172.21.15.137:0/31975649 conn(0x7faa780f0b70 msgr2=0x7faa780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 -- 172.21.15.137:0/31975649 shutdown_connections 2023-11-15T05:59:07.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.859+0000 7faa7ff26700 1 -- 172.21.15.137:0/31975649 wait complete. 2023-11-15T05:59:07.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7ff26700 1 Processor -- start 2023-11-15T05:59:07.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7ff26700 1 -- start start 2023-11-15T05:59:07.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7ff26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:07.863 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7ff26700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa78106c20 con 0x7faa780f5410 2023-11-15T05:59:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7dcc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7dcc2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:57432/0 (socket says 172.21.15.137:57432) 2023-11-15T05:59:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7dcc2700 1 -- 172.21.15.137:0/67448946 learned_addr learned my addr 172.21.15.137:0/67448946 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7dcc2700 1 -- 172.21.15.137:0/67448946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa68005040 con 0x7faa780f5410 2023-11-15T05:59:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.860+0000 7faa7dcc2700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7faa68013510 tx=0x7faa6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa68009320 con 0x7faa780f5410 2023-11-15T05:59:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa78106e20 con 0x7faa780f5410 2023-11-15T05:59:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa68013f00 con 0x7faa780f5410 2023-11-15T05:59:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa68014d90 con 0x7faa780f5410 2023-11-15T05:59:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa781072b0 con 0x7faa780f5410 2023-11-15T05:59:07.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa78045ad0 con 0x7faa780f5410 2023-11-15T05:59:07.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.861+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa680095f0 con 0x7faa780f5410 2023-11-15T05:59:07.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.862+0000 7faa6e7fc700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 0x7faa64032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:07.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.862+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa68040ff0 con 0x7faa780f5410 2023-11-15T05:59:07.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.862+0000 7faa7d4c1700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 0x7faa64032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:07.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.862+0000 7faa7d4c1700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 0x7faa64032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faa740096a0 tx=0x7faa74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:07.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:07.866+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa6806d2b0 con 0x7faa780f5410 2023-11-15T05:59:08.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.071+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa780fae60 con 0x7faa780f5410 2023-11-15T05:59:08.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.072+0000 7faa6e7fc700 1 -- 172.21.15.137:0/67448946 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7faa68014420 con 0x7faa780f5410 2023-11-15T05:59:08.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:08.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:08.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 msgr2=0x7faa64032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:08.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 0x7faa64032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faa740096a0 tx=0x7faa74006b40).stop 2023-11-15T05:59:08.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 msgr2=0x7faa78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:08.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7faa68013510 tx=0x7faa6800b040).stop 2023-11-15T05:59:08.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 shutdown_connections 2023-11-15T05:59:08.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7faa6402ffb0 0x7faa64032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:08.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 --2- 172.21.15.137:0/67448946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faa780f5410 0x7faa78106730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:08.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 >> 172.21.15.137:0/67448946 conn(0x7faa780f0b70 msgr2=0x7faa780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:08.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 shutdown_connections 2023-11-15T05:59:08.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:08.073+0000 7faa7ff26700 1 -- 172.21.15.137:0/67448946 wait complete. 2023-11-15T05:59:08.076 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:07.006060+0000 mgr.smithi003.cldzke (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:09.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:08.072156+0000 mon.smithi003 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.137:0/67448946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:09.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:09.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:10.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:09.006319+0000 mgr.smithi003.cldzke (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:10.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- 172.21.15.137:0/2639984285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 msgr2=0x7fa7e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:10.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 --2- 172.21.15.137:0/2639984285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa7d4009fb0 tx=0x7fa7d4009b30).stop 2023-11-15T05:59:10.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- 172.21.15.137:0/2639984285 shutdown_connections 2023-11-15T05:59:10.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 --2- 172.21.15.137:0/2639984285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:10.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- 172.21.15.137:0/2639984285 >> 172.21.15.137:0/2639984285 conn(0x7fa7e40f0b70 msgr2=0x7fa7e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:10.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- 172.21.15.137:0/2639984285 shutdown_connections 2023-11-15T05:59:10.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- 172.21.15.137:0/2639984285 wait complete. 2023-11-15T05:59:10.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 Processor -- start 2023-11-15T05:59:10.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- start start 2023-11-15T05:59:10.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:10.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.699+0000 7fa7ea512700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40fb820 con 0x7fa7e40f5410 2023-11-15T05:59:10.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:10.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42372/0 (socket says 172.21.15.137:42372) 2023-11-15T05:59:10.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e3fff700 1 -- 172.21.15.137:0/1602106096 learned_addr learned my addr 172.21.15.137:0/1602106096 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e3fff700 1 -- 172.21.15.137:0/1602106096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4005040 con 0x7fa7e40f5410 2023-11-15T05:59:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e3fff700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa7d4013510 tx=0x7fa7d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.700+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7d4009320 con 0x7fa7e40f5410 2023-11-15T05:59:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7d4013f00 con 0x7fa7e40f5410 2023-11-15T05:59:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e40fb9c0 con 0x7fa7e40f5410 2023-11-15T05:59:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7d4014d90 con 0x7fa7e40f5410 2023-11-15T05:59:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e40fbe50 con 0x7fa7e40f5410 2023-11-15T05:59:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7d40095f0 con 0x7fa7e40f5410 2023-11-15T05:59:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.701+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e4059100 con 0x7fa7e40f5410 2023-11-15T05:59:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.702+0000 7fa7e0ff9700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 0x7fa7cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:10.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.702+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7d40410c0 con 0x7fa7e40f5410 2023-11-15T05:59:10.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.702+0000 7fa7e37fe700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 0x7fa7cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:10.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.702+0000 7fa7e37fe700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 0x7fa7cc032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa7d80096a0 tx=0x7fa7d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:10.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.706+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7d401d7f0 con 0x7fa7e40f5410 2023-11-15T05:59:10.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.912+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7e40fae60 con 0x7fa7e40f5410 2023-11-15T05:59:10.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.913+0000 7fa7e0ff9700 1 -- 172.21.15.137:0/1602106096 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa7d4010230 con 0x7fa7e40f5410 2023-11-15T05:59:10.914 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:10.914 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:10.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 msgr2=0x7fa7cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:10.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 0x7fa7cc032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa7d80096a0 tx=0x7fa7d8006b40).stop 2023-11-15T05:59:10.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 msgr2=0x7fa7e40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:10.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa7d4013510 tx=0x7fa7d400b040).stop 2023-11-15T05:59:10.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 shutdown_connections 2023-11-15T05:59:10.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7cc02ff60 0x7fa7cc032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:10.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 --2- 172.21.15.137:0/1602106096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7e40f5410 0x7fa7e40fd3a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:10.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 >> 172.21.15.137:0/1602106096 conn(0x7fa7e40f0b70 msgr2=0x7fa7e405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:10.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 shutdown_connections 2023-11-15T05:59:10.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:10.915+0000 7fa7ea512700 1 -- 172.21.15.137:0/1602106096 wait complete. 2023-11-15T05:59:10.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:10.913336+0000 mon.smithi003 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.137:0/1602106096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:11.006662+0000 mgr.smithi003.cldzke (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:12.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:12.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:13.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.505+0000 7f77177c6700 1 -- 172.21.15.137:0/3817339593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 msgr2=0x7f77100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:13.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.505+0000 7f77177c6700 1 --2- 172.21.15.137:0/3817339593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f77100f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f7700009fb0 tx=0x7f7700009b30).stop 2023-11-15T05:59:13.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.505+0000 7f77177c6700 1 -- 172.21.15.137:0/3817339593 shutdown_connections 2023-11-15T05:59:13.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.505+0000 7f77177c6700 1 --2- 172.21.15.137:0/3817339593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f77100f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:13.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.505+0000 7f77177c6700 1 -- 172.21.15.137:0/3817339593 >> 172.21.15.137:0/3817339593 conn(0x7f77100f0b70 msgr2=0x7f77100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:13.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 -- 172.21.15.137:0/3817339593 shutdown_connections 2023-11-15T05:59:13.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 -- 172.21.15.137:0/3817339593 wait complete. 2023-11-15T05:59:13.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 Processor -- start 2023-11-15T05:59:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 -- start start 2023-11-15T05:59:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.506+0000 7f77177c6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7710106c50 con 0x7f77100f5410 2023-11-15T05:59:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7715562700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7715562700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42384/0 (socket says 172.21.15.137:42384) 2023-11-15T05:59:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7715562700 1 -- 172.21.15.137:0/159344828 learned_addr learned my addr 172.21.15.137:0/159344828 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7715562700 1 -- 172.21.15.137:0/159344828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7700005040 con 0x7f77100f5410 2023-11-15T05:59:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7715562700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7700009f80 tx=0x7f770000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7700009320 con 0x7f77100f5410 2023-11-15T05:59:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7700013ec0 con 0x7f77100f5410 2023-11-15T05:59:13.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7710106e50 con 0x7f77100f5410 2023-11-15T05:59:13.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f770001ece0 con 0x7f77100f5410 2023-11-15T05:59:13.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.507+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77101072e0 con 0x7f77100f5410 2023-11-15T05:59:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.508+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7710045ad0 con 0x7f77100f5410 2023-11-15T05:59:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.508+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f770001e420 con 0x7f77100f5410 2023-11-15T05:59:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.508+0000 7f7705ffb700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 0x7f76fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.508+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7700040cf0 con 0x7f77100f5410 2023-11-15T05:59:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.509+0000 7f7714d61700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 0x7f76fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.509+0000 7f7714d61700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 0x7f76fc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f770c0096a0 tx=0x7f770c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.513+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f770001e730 con 0x7f77100f5410 2023-11-15T05:59:13.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.716+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7710108190 con 0x7f77100f5410 2023-11-15T05:59:13.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.717+0000 7f7705ffb700 1 -- 172.21.15.137:0/159344828 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7700004490 con 0x7f77100f5410 2023-11-15T05:59:13.718 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:13.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:13.719 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.718+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 msgr2=0x7f76fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:13.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.718+0000 7f77177c6700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 0x7f76fc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f770c0096a0 tx=0x7f770c006b40).stop 2023-11-15T05:59:13.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.718+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 msgr2=0x7f7710106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:13.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.718+0000 7f77177c6700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f7700009f80 tx=0x7f770000b040).stop 2023-11-15T05:59:13.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 shutdown_connections 2023-11-15T05:59:13.720 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f76fc02ff60 0x7f76fc032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:13.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 --2- 172.21.15.137:0/159344828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77100f5410 0x7f7710106760 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:13.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 >> 172.21.15.137:0/159344828 conn(0x7f77100f0b70 msgr2=0x7f771005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:13.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 shutdown_connections 2023-11-15T05:59:13.721 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:13.719+0000 7f77177c6700 1 -- 172.21.15.137:0/159344828 wait complete. 2023-11-15T05:59:13.722 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:14.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:13.007045+0000 mgr.smithi003.cldzke (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:14.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:13.717111+0000 mon.smithi003 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.137:0/159344828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:15.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:15.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:16.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 -- 172.21.15.137:0/347733577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 msgr2=0x7fb3940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:16.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 --2- 172.21.15.137:0/347733577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb3940f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb388009fb0 tx=0x7fb388009b30).stop 2023-11-15T05:59:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 -- 172.21.15.137:0/347733577 shutdown_connections 2023-11-15T05:59:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 --2- 172.21.15.137:0/347733577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb3940f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 -- 172.21.15.137:0/347733577 >> 172.21.15.137:0/347733577 conn(0x7fb3940f0b70 msgr2=0x7fb3940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 -- 172.21.15.137:0/347733577 shutdown_connections 2023-11-15T05:59:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.250+0000 7fb399160700 1 -- 172.21.15.137:0/347733577 wait complete. 2023-11-15T05:59:16.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.251+0000 7fb399160700 1 Processor -- start 2023-11-15T05:59:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.251+0000 7fb399160700 1 -- start start 2023-11-15T05:59:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.251+0000 7fb399160700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.251+0000 7fb399160700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb394106c00 con 0x7fb3940f5410 2023-11-15T05:59:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb392d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb392d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42400/0 (socket says 172.21.15.137:42400) 2023-11-15T05:59:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb392d9d700 1 -- 172.21.15.137:0/3718391718 learned_addr learned my addr 172.21.15.137:0/3718391718 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb392d9d700 1 -- 172.21.15.137:0/3718391718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388005040 con 0x7fb3940f5410 2023-11-15T05:59:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb392d9d700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb388013510 tx=0x7fb38800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb388009320 con 0x7fb3940f5410 2023-11-15T05:59:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb394106e00 con 0x7fb3940f5410 2023-11-15T05:59:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb388013f00 con 0x7fb3940f5410 2023-11-15T05:59:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb388014d90 con 0x7fb3940f5410 2023-11-15T05:59:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.252+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb394107290 con 0x7fb3940f5410 2023-11-15T05:59:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.253+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3880095f0 con 0x7fb3940f5410 2023-11-15T05:59:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.253+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb394045ad0 con 0x7fb3940f5410 2023-11-15T05:59:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.253+0000 7fb37b7fe700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.253+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb388041000 con 0x7fb3940f5410 2023-11-15T05:59:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.254+0000 7fb39259c700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.254+0000 7fb39259c700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 0x7fb37c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fb3840096a0 tx=0x7fb384006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:16.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.258+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb38801d7f0 con 0x7fb3940f5410 2023-11-15T05:59:16.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:15.007379+0000 mgr.smithi003.cldzke (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:16.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.464+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3940fae60 con 0x7fb3940f5410 2023-11-15T05:59:16.466 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.465+0000 7fb37b7fe700 1 -- 172.21.15.137:0/3718391718 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb388010230 con 0x7fb3940f5410 2023-11-15T05:59:16.466 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:16.467 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:16.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 msgr2=0x7fb37c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:16.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 0x7fb37c032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fb3840096a0 tx=0x7fb384006b40).stop 2023-11-15T05:59:16.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 msgr2=0x7fb394106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:16.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb388013510 tx=0x7fb38800b040).stop 2023-11-15T05:59:16.468 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 shutdown_connections 2023-11-15T05:59:16.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:16.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 --2- 172.21.15.137:0/3718391718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3940f5410 0x7fb394106710 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:16.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.466+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 >> 172.21.15.137:0/3718391718 conn(0x7fb3940f0b70 msgr2=0x7fb39405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:16.469 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.467+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 shutdown_connections 2023-11-15T05:59:16.470 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:16.467+0000 7fb399160700 1 -- 172.21.15.137:0/3718391718 wait complete. 2023-11-15T05:59:16.470 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:17.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:17 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:16.465330+0000 mon.smithi003 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.137:0/3718391718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:17.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:17.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:18.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:17.007734+0000 mgr.smithi003.cldzke (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:19.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 -- 172.21.15.137:0/4170035497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 msgr2=0x7f2d940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:19.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4170035497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d940f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f2d84009fb0 tx=0x7f2d84009b30).stop 2023-11-15T05:59:19.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 -- 172.21.15.137:0/4170035497 shutdown_connections 2023-11-15T05:59:19.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4170035497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d940f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:19.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 -- 172.21.15.137:0/4170035497 >> 172.21.15.137:0/4170035497 conn(0x7f2d940f0b70 msgr2=0x7f2d940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:19.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 -- 172.21.15.137:0/4170035497 shutdown_connections 2023-11-15T05:59:19.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.054+0000 7f2d9a181700 1 -- 172.21.15.137:0/4170035497 wait complete. 2023-11-15T05:59:19.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d9a181700 1 Processor -- start 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d9a181700 1 -- start start 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d9a181700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d9a181700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d94105b20 con 0x7f2d940f5410 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d937fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42408/0 (socket says 172.21.15.137:42408) 2023-11-15T05:59:19.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d937fe700 1 -- 172.21.15.137:0/4168975907 learned_addr learned my addr 172.21.15.137:0/4168975907 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:19.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.055+0000 7f2d937fe700 1 -- 172.21.15.137:0/4168975907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84005040 con 0x7f2d940f5410 2023-11-15T05:59:19.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d937fe700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2d84013510 tx=0x7f2d8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:19.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d84009320 con 0x7f2d940f5410 2023-11-15T05:59:19.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d94105d20 con 0x7f2d940f5410 2023-11-15T05:59:19.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d84013f00 con 0x7f2d940f5410 2023-11-15T05:59:19.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d84014d90 con 0x7f2d940f5410 2023-11-15T05:59:19.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.056+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d941061b0 con 0x7f2d940f5410 2023-11-15T05:59:19.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d840095f0 con 0x7f2d940f5410 2023-11-15T05:59:19.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d94045ad0 con 0x7f2d940f5410 2023-11-15T05:59:19.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d7bfff700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 0x7f2d7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d840410a0 con 0x7f2d940f5410 2023-11-15T05:59:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d92ffd700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 0x7f2d7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.057+0000 7f2d92ffd700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 0x7f2d7c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2d880096a0 tx=0x7f2d88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:19.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.062+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d8401d7f0 con 0x7f2d940f5410 2023-11-15T05:59:19.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.269+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d940fae60 con 0x7f2d940f5410 2023-11-15T05:59:19.271 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.270+0000 7f2d7bfff700 1 -- 172.21.15.137:0/4168975907 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2d84010230 con 0x7f2d940f5410 2023-11-15T05:59:19.271 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:19.272 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:19.273 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 msgr2=0x7f2d7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:19.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 0x7f2d7c032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2d880096a0 tx=0x7f2d88006b40).stop 2023-11-15T05:59:19.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 msgr2=0x7f2d94105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:19.274 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f2d84013510 tx=0x7f2d8400b040).stop 2023-11-15T05:59:19.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 shutdown_connections 2023-11-15T05:59:19.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2d7c02ff60 0x7f2d7c032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:19.275 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 --2- 172.21.15.137:0/4168975907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d940f5410 0x7f2d94105630 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:19.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 >> 172.21.15.137:0/4168975907 conn(0x7f2d940f0b70 msgr2=0x7f2d940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:19.276 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 shutdown_connections 2023-11-15T05:59:19.277 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:19.271+0000 7f2d9a181700 1 -- 172.21.15.137:0/4168975907 wait complete. 2023-11-15T05:59:19.278 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:20.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:19.008021+0000 mgr.smithi003.cldzke (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:19.270146+0000 mon.smithi003 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.137:0/4168975907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:20.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:20.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.886+0000 7fe423f06700 1 -- 172.21.15.137:0/728874653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 msgr2=0x7fe41c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.886+0000 7fe423f06700 1 --2- 172.21.15.137:0/728874653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c0f9150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fe40c009fb0 tx=0x7fe40c009b30).stop 2023-11-15T05:59:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.886+0000 7fe423f06700 1 -- 172.21.15.137:0/728874653 shutdown_connections 2023-11-15T05:59:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.886+0000 7fe423f06700 1 --2- 172.21.15.137:0/728874653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c0f9150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:21.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.886+0000 7fe423f06700 1 -- 172.21.15.137:0/728874653 >> 172.21.15.137:0/728874653 conn(0x7fe41c0f0b70 msgr2=0x7fe41c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:21.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 -- 172.21.15.137:0/728874653 shutdown_connections 2023-11-15T05:59:21.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 -- 172.21.15.137:0/728874653 wait complete. 2023-11-15T05:59:21.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 Processor -- start 2023-11-15T05:59:21.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 -- start start 2023-11-15T05:59:21.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:21.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.887+0000 7fe423f06700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c106bd0 con 0x7fe41c060220 2023-11-15T05:59:21.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe421ca2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:21.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe421ca2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47932/0 (socket says 172.21.15.137:47932) 2023-11-15T05:59:21.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe421ca2700 1 -- 172.21.15.137:0/1363066446 learned_addr learned my addr 172.21.15.137:0/1363066446 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:21.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe421ca2700 1 -- 172.21.15.137:0/1363066446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c005040 con 0x7fe41c060220 2023-11-15T05:59:21.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe421ca2700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fe40c013510 tx=0x7fe40c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:21.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.888+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40c009320 con 0x7fe41c060220 2023-11-15T05:59:21.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c106dd0 con 0x7fe41c060220 2023-11-15T05:59:21.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe40c013f00 con 0x7fe41c060220 2023-11-15T05:59:21.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40c014dc0 con 0x7fe41c060220 2023-11-15T05:59:21.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c107260 con 0x7fe41c060220 2023-11-15T05:59:21.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe40c0095f0 con 0x7fe41c060220 2023-11-15T05:59:21.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c045ad0 con 0x7fe41c060220 2023-11-15T05:59:21.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.889+0000 7fe4127fc700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 0x7fe408032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:21.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.890+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe40c0410c0 con 0x7fe41c060220 2023-11-15T05:59:21.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.890+0000 7fe4214a1700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 0x7fe408032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:21.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.890+0000 7fe4214a1700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 0x7fe408032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe4180096a0 tx=0x7fe418006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:21.897 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:21.894+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe40c01c790 con 0x7fe41c060220 2023-11-15T05:59:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:21.008365+0000 mgr.smithi003.cldzke (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:22.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.099+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe41c108110 con 0x7fe41c060220 2023-11-15T05:59:22.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.100+0000 7fe4127fc700 1 -- 172.21.15.137:0/1363066446 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe40c010230 con 0x7fe41c060220 2023-11-15T05:59:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:22.101 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:22.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 msgr2=0x7fe408032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 0x7fe408032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe4180096a0 tx=0x7fe418006b40).stop 2023-11-15T05:59:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 msgr2=0x7fe41c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fe40c013510 tx=0x7fe40c00b040).stop 2023-11-15T05:59:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 shutdown_connections 2023-11-15T05:59:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe40802ff60 0x7fe408032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:22.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 --2- 172.21.15.137:0/1363066446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe41c060220 0x7fe41c1066e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:22.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 >> 172.21.15.137:0/1363066446 conn(0x7fe41c0f0b70 msgr2=0x7fe41c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:22.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 shutdown_connections 2023-11-15T05:59:22.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:22.101+0000 7fe423f06700 1 -- 172.21.15.137:0/1363066446 wait complete. 2023-11-15T05:59:22.104 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:22.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:22.100125+0000 mon.smithi003 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.137:0/1363066446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:23.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:23.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:23.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:23.008673+0000 mgr.smithi003.cldzke (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:24.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 -- 172.21.15.137:0/700133368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 msgr2=0x7f02a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:24.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 --2- 172.21.15.137:0/700133368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a80f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f029c009fb0 tx=0x7f029c009b30).stop 2023-11-15T05:59:24.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 -- 172.21.15.137:0/700133368 shutdown_connections 2023-11-15T05:59:24.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 --2- 172.21.15.137:0/700133368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a80f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:24.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 -- 172.21.15.137:0/700133368 >> 172.21.15.137:0/700133368 conn(0x7f02a80f1c70 msgr2=0x7f02a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:24.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 -- 172.21.15.137:0/700133368 shutdown_connections 2023-11-15T05:59:24.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.648+0000 7f02af1c8700 1 -- 172.21.15.137:0/700133368 wait complete. 2023-11-15T05:59:24.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02af1c8700 1 Processor -- start 2023-11-15T05:59:24.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02af1c8700 1 -- start start 2023-11-15T05:59:24.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02af1c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:24.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02af1c8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8069e20 con 0x7f02a80f6510 2023-11-15T05:59:24.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02acf64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:24.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02acf64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47956/0 (socket says 172.21.15.137:47956) 2023-11-15T05:59:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.649+0000 7f02acf64700 1 -- 172.21.15.137:0/754901588 learned_addr learned my addr 172.21.15.137:0/754901588 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02acf64700 1 -- 172.21.15.137:0/754901588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c005040 con 0x7f02a80f6510 2023-11-15T05:59:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02acf64700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f029c013510 tx=0x7f029c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029c009320 con 0x7f02a80f6510 2023-11-15T05:59:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f029c013f00 con 0x7f02a80f6510 2023-11-15T05:59:24.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8069fc0 con 0x7f02a80f6510 2023-11-15T05:59:24.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f029c014d90 con 0x7f02a80f6510 2023-11-15T05:59:24.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.650+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a806a450 con 0x7f02a80f6510 2023-11-15T05:59:24.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.651+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f029c0095f0 con 0x7f02a80f6510 2023-11-15T05:59:24.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.651+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a8059100 con 0x7f02a80f6510 2023-11-15T05:59:24.657 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.651+0000 7f02a57fa700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 0x7f0298032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:24.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.651+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f029c0410a0 con 0x7f02a80f6510 2023-11-15T05:59:24.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.651+0000 7f02a7fff700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 0x7f0298032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:24.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.652+0000 7f02a7fff700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 0x7f0298032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f02940096a0 tx=0x7f0294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:24.658 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.656+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f029c01d7f0 con 0x7f02a80f6510 2023-11-15T05:59:24.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.864+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02a80fbf60 con 0x7f02a80f6510 2023-11-15T05:59:24.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.864+0000 7f02a57fa700 1 -- 172.21.15.137:0/754901588 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f029c010230 con 0x7f02a80f6510 2023-11-15T05:59:24.865 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:24.865 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:24.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 msgr2=0x7f0298032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 0x7f0298032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f02940096a0 tx=0x7f0294006b40).stop 2023-11-15T05:59:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 msgr2=0x7f02a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f029c013510 tx=0x7f029c00b040).stop 2023-11-15T05:59:24.867 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 shutdown_connections 2023-11-15T05:59:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f029802ff60 0x7f0298032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 --2- 172.21.15.137:0/754901588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a80f6510 0x7f02a806b9a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 >> 172.21.15.137:0/754901588 conn(0x7f02a80f1c70 msgr2=0x7f02a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 shutdown_connections 2023-11-15T05:59:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:24.866+0000 7f02af1c8700 1 -- 172.21.15.137:0/754901588 wait complete. 2023-11-15T05:59:24.869 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:25.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:24.864814+0000 mon.smithi003 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.137:0/754901588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:26.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:25.009041+0000 mgr.smithi003.cldzke (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:26.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:26.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:27.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3191883051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 msgr2=0x7ff8f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:27.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3191883051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f90f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7ff8e8009fb0 tx=0x7ff8e8009b30).stop 2023-11-15T05:59:27.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3191883051 shutdown_connections 2023-11-15T05:59:27.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3191883051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f90f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:27.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3191883051 >> 172.21.15.137:0/3191883051 conn(0x7ff8f80f0b70 msgr2=0x7ff8f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:27.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3191883051 shutdown_connections 2023-11-15T05:59:27.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.484+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3191883051 wait complete. 2023-11-15T05:59:27.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8ff9b7700 1 Processor -- start 2023-11-15T05:59:27.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8ff9b7700 1 -- start start 2023-11-15T05:59:27.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8ff9b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:27.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8ff9b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f80f4ff0 con 0x7ff8f8060280 2023-11-15T05:59:27.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8fd753700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:27.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8fd753700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47970/0 (socket says 172.21.15.137:47970) 2023-11-15T05:59:27.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8fd753700 1 -- 172.21.15.137:0/3920533859 learned_addr learned my addr 172.21.15.137:0/3920533859 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:27.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.485+0000 7ff8fd753700 1 -- 172.21.15.137:0/3920533859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8005040 con 0x7ff8f8060280 2023-11-15T05:59:27.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8fd753700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff8e8013510 tx=0x7ff8e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:27.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8e8009320 con 0x7ff8f8060280 2023-11-15T05:59:27.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f80f5190 con 0x7ff8f8060280 2023-11-15T05:59:27.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8e8013f00 con 0x7ff8f8060280 2023-11-15T05:59:27.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8e8014dc0 con 0x7ff8f8060280 2023-11-15T05:59:27.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.486+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f80f5620 con 0x7ff8f8060280 2023-11-15T05:59:27.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.487+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8e8014420 con 0x7ff8f8060280 2023-11-15T05:59:27.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.487+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f8059100 con 0x7ff8f8060280 2023-11-15T05:59:27.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.487+0000 7ff8edffb700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:27.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.487+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8e80410f0 con 0x7ff8f8060280 2023-11-15T05:59:27.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.487+0000 7ff8fcf52700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:27.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.488+0000 7ff8fcf52700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:27.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.492+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8e80095f0 con 0x7ff8f8060280 2023-11-15T05:59:27.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.694+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8f80fbf50 con 0x7ff8f8060280 2023-11-15T05:59:27.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.695+0000 7ff8edffb700 1 -- 172.21.15.137:0/3920533859 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff8e801cb10 con 0x7ff8f8060280 2023-11-15T05:59:27.697 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:27.698 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:27.698 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 msgr2=0x7ff8e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:27.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).stop 2023-11-15T05:59:27.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 msgr2=0x7ff8f80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:27.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff8e8013510 tx=0x7ff8e800b040).stop 2023-11-15T05:59:27.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 shutdown_connections 2023-11-15T05:59:27.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:27.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 --2- 172.21.15.137:0/3920533859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8f8060280 0x7ff8f80f6b70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:27.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 >> 172.21.15.137:0/3920533859 conn(0x7ff8f80f0b70 msgr2=0x7ff8f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:27.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 shutdown_connections 2023-11-15T05:59:27.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:27.697+0000 7ff8ff9b7700 1 -- 172.21.15.137:0/3920533859 wait complete. 2023-11-15T05:59:27.701 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:27.009397+0000 mgr.smithi003.cldzke (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:27.696758+0000 mon.smithi003 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.137:0/3920533859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:29.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:29.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:30.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 -- 172.21.15.137:0/3960402824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 msgr2=0x7f98e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:30.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 --2- 172.21.15.137:0/3960402824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e00f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f98d4009fb0 tx=0x7f98d4009b30).stop 2023-11-15T05:59:30.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 -- 172.21.15.137:0/3960402824 shutdown_connections 2023-11-15T05:59:30.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 --2- 172.21.15.137:0/3960402824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e00f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:30.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 -- 172.21.15.137:0/3960402824 >> 172.21.15.137:0/3960402824 conn(0x7f98e00f1c70 msgr2=0x7f98e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:30.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 -- 172.21.15.137:0/3960402824 shutdown_connections 2023-11-15T05:59:30.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.294+0000 7f98e48f7700 1 -- 172.21.15.137:0/3960402824 wait complete. 2023-11-15T05:59:30.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98e48f7700 1 Processor -- start 2023-11-15T05:59:30.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98e48f7700 1 -- start start 2023-11-15T05:59:30.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98e48f7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:30.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98e48f7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0069e20 con 0x7f98e00f6510 2023-11-15T05:59:30.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:30.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34392/0 (socket says 172.21.15.137:34392) 2023-11-15T05:59:30.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.295+0000 7f98de59c700 1 -- 172.21.15.137:0/1205602092 learned_addr learned my addr 172.21.15.137:0/1205602092 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:30.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98de59c700 1 -- 172.21.15.137:0/1205602092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4005040 con 0x7f98e00f6510 2023-11-15T05:59:30.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98de59c700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f98d4014000 tx=0x7f98d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:30.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d40095f0 con 0x7f98e00f6510 2023-11-15T05:59:30.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e0069fc0 con 0x7f98e00f6510 2023-11-15T05:59:30.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98d4007e70 con 0x7f98e00f6510 2023-11-15T05:59:30.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d4014ea0 con 0x7f98e00f6510 2023-11-15T05:59:30.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.296+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e006a450 con 0x7f98e00f6510 2023-11-15T05:59:30.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.297+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98d4025070 con 0x7f98e00f6510 2023-11-15T05:59:30.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.297+0000 7f98ceffd700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 0x7f98c8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:30.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.297+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98d4040cd0 con 0x7f98e00f6510 2023-11-15T05:59:30.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.297+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0059100 con 0x7f98e00f6510 2023-11-15T05:59:30.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.297+0000 7f98ddd9b700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 0x7f98c8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:30.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.298+0000 7f98ddd9b700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 0x7f98c8032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f98d00096a0 tx=0x7f98d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:30.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.302+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98d4026350 con 0x7f98e00f6510 2023-11-15T05:59:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:29.009745+0000 mgr.smithi003.cldzke (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:30.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.506+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98e0045ad0 con 0x7f98e00f6510 2023-11-15T05:59:30.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.507+0000 7f98ceffd700 1 -- 172.21.15.137:0/1205602092 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f98d4012330 con 0x7f98e00f6510 2023-11-15T05:59:30.509 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:30.509 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:30.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 msgr2=0x7f98c8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:30.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 0x7f98c8032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f98d00096a0 tx=0x7f98d0006b40).stop 2023-11-15T05:59:30.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 msgr2=0x7f98e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:30.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f98d4014000 tx=0x7f98d400b040).stop 2023-11-15T05:59:30.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 shutdown_connections 2023-11-15T05:59:30.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f98c802ffb0 0x7f98c8032470 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:30.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 --2- 172.21.15.137:0/1205602092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98e00f6510 0x7f98e006b9a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:30.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 >> 172.21.15.137:0/1205602092 conn(0x7f98e00f1c70 msgr2=0x7f98e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:30.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 shutdown_connections 2023-11-15T05:59:30.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:30.509+0000 7f98e48f7700 1 -- 172.21.15.137:0/1205602092 wait complete. 2023-11-15T05:59:30.513 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:31.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:30.508536+0000 mon.smithi003 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.137:0/1205602092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:31.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:31.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:32.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:31.010027+0000 mgr.smithi003.cldzke (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:33.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 -- 172.21.15.137:0/419682876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 msgr2=0x7f95f00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:33.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/419682876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f00f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f95e0009fb0 tx=0x7f95e0009b30).stop 2023-11-15T05:59:33.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 -- 172.21.15.137:0/419682876 shutdown_connections 2023-11-15T05:59:33.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/419682876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f00f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 -- 172.21.15.137:0/419682876 >> 172.21.15.137:0/419682876 conn(0x7f95f00f0b70 msgr2=0x7f95f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 -- 172.21.15.137:0/419682876 shutdown_connections 2023-11-15T05:59:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.013+0000 7f95f5cbb700 1 -- 172.21.15.137:0/419682876 wait complete. 2023-11-15T05:59:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95f5cbb700 1 Processor -- start 2023-11-15T05:59:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95f5cbb700 1 -- start start 2023-11-15T05:59:33.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95f5cbb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:33.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95f5cbb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f0108dd0 con 0x7f95f0060280 2023-11-15T05:59:33.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:33.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95ef7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34414/0 (socket says 172.21.15.137:34414) 2023-11-15T05:59:33.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.014+0000 7f95ef7fe700 1 -- 172.21.15.137:0/4116401866 learned_addr learned my addr 172.21.15.137:0/4116401866 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:33.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95ef7fe700 1 -- 172.21.15.137:0/4116401866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0005040 con 0x7f95f0060280 2023-11-15T05:59:33.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95ef7fe700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f95e0013510 tx=0x7f95e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:33.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e0009320 con 0x7f95f0060280 2023-11-15T05:59:33.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f0108fd0 con 0x7f95f0060280 2023-11-15T05:59:33.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95e0013f00 con 0x7f95f0060280 2023-11-15T05:59:33.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e0014dc0 con 0x7f95f0060280 2023-11-15T05:59:33.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.015+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f0109460 con 0x7f95f0060280 2023-11-15T05:59:33.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.016+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95e0014420 con 0x7f95f0060280 2023-11-15T05:59:33.023 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.016+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f0045ad0 con 0x7f95f0060280 2023-11-15T05:59:33.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.016+0000 7f95e7fff700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:33.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.016+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95e0041040 con 0x7f95f0060280 2023-11-15T05:59:33.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.016+0000 7f95eeffd700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:33.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.017+0000 7f95eeffd700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 0x7f95dc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f95d80096a0 tx=0x7f95d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:33.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.021+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95e00095f0 con 0x7f95f0060280 2023-11-15T05:59:33.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.227+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95f010a2b0 con 0x7f95f0060280 2023-11-15T05:59:33.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.228+0000 7f95e7fff700 1 -- 172.21.15.137:0/4116401866 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f95e001cb10 con 0x7f95f0060280 2023-11-15T05:59:33.230 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:33.230 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:33.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 msgr2=0x7f95dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:33.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 0x7f95dc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f95d80096a0 tx=0x7f95d8006b40).stop 2023-11-15T05:59:33.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 msgr2=0x7f95f01088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:33.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f95e0013510 tx=0x7f95e000b040).stop 2023-11-15T05:59:33.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 shutdown_connections 2023-11-15T05:59:33.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:33.232 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 --2- 172.21.15.137:0/4116401866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95f0060280 0x7f95f01088e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 >> 172.21.15.137:0/4116401866 conn(0x7f95f00f0b70 msgr2=0x7f95f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 shutdown_connections 2023-11-15T05:59:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:33.230+0000 7f95f5cbb700 1 -- 172.21.15.137:0/4116401866 wait complete. 2023-11-15T05:59:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:33.010281+0000 mgr.smithi003.cldzke (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:33.229554+0000 mon.smithi003 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.137:0/4116401866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:34.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:34.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:35.807 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.804+0000 7f1295fc4700 1 -- 172.21.15.137:0/3310390911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 msgr2=0x7f12900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:36.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.804+0000 7f1295fc4700 1 --2- 172.21.15.137:0/3310390911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f12900f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f1284009fb0 tx=0x7f1284009b30).stop 2023-11-15T05:59:36.010 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.804+0000 7f1295fc4700 1 -- 172.21.15.137:0/3310390911 shutdown_connections 2023-11-15T05:59:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.804+0000 7f1295fc4700 1 --2- 172.21.15.137:0/3310390911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f12900f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.804+0000 7f1295fc4700 1 -- 172.21.15.137:0/3310390911 >> 172.21.15.137:0/3310390911 conn(0x7f12900f0b70 msgr2=0x7f12900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 -- 172.21.15.137:0/3310390911 shutdown_connections 2023-11-15T05:59:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 -- 172.21.15.137:0/3310390911 wait complete. 2023-11-15T05:59:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 Processor -- start 2023-11-15T05:59:36.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 -- start start 2023-11-15T05:59:36.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:36.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.805+0000 7f1295fc4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1290060240 con 0x7f12900f5410 2023-11-15T05:59:36.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f128f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:36.012 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f128f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34420/0 (socket says 172.21.15.137:34420) 2023-11-15T05:59:36.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f128f7fe700 1 -- 172.21.15.137:0/1738436195 learned_addr learned my addr 172.21.15.137:0/1738436195 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:36.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f128f7fe700 1 -- 172.21.15.137:0/1738436195 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284005040 con 0x7f12900f5410 2023-11-15T05:59:36.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f128f7fe700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1284014000 tx=0x7f128400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:36.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12840095f0 con 0x7f12900f5410 2023-11-15T05:59:36.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1290060440 con 0x7f12900f5410 2023-11-15T05:59:36.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1284007e70 con 0x7f12900f5410 2023-11-15T05:59:36.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1290109ae0 con 0x7f12900f5410 2023-11-15T05:59:36.017 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.806+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1284014ea0 con 0x7f12900f5410 2023-11-15T05:59:36.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.807+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1284025070 con 0x7f12900f5410 2023-11-15T05:59:36.018 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.807+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1290045ad0 con 0x7f12900f5410 2023-11-15T05:59:36.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.807+0000 7f1277fff700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 0x7f1278032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:36.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.807+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12840409b0 con 0x7f12900f5410 2023-11-15T05:59:36.019 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.808+0000 7f128effd700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 0x7f1278032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:36.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.808+0000 7f128effd700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 0x7f1278032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f12800096a0 tx=0x7f1280006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:36.020 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:35.812+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f128400f4e0 con 0x7f12900f5410 2023-11-15T05:59:36.021 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.018+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12900fae60 con 0x7f12900f5410 2023-11-15T05:59:36.022 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.018+0000 7f1277fff700 1 -- 172.21.15.137:0/1738436195 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f128400f300 con 0x7f12900f5410 2023-11-15T05:59:36.022 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:36.023 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:36.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 msgr2=0x7f1278032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:36.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 0x7f1278032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f12800096a0 tx=0x7f1280006b40).stop 2023-11-15T05:59:36.024 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 msgr2=0x7f1290109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:36.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1284014000 tx=0x7f128400b040).stop 2023-11-15T05:59:36.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 shutdown_connections 2023-11-15T05:59:36.025 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f127802ff60 0x7f1278032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:36.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 --2- 172.21.15.137:0/1738436195 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12900f5410 0x7f1290109430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:36.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 >> 172.21.15.137:0/1738436195 conn(0x7f12900f0b70 msgr2=0x7f12900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:36.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 shutdown_connections 2023-11-15T05:59:36.027 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:36.020+0000 7f1295fc4700 1 -- 172.21.15.137:0/1738436195 wait complete. 2023-11-15T05:59:36.027 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:35.010604+0000 mgr.smithi003.cldzke (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:35.114511+0000 mon.smithi003 (mon.0) 222 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T05:59:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:35.117566+0000 mon.smithi003 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T05:59:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:36.019896+0000 mon.smithi003 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.137:0/1738436195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:37.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:37.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:37.010908+0000 mgr.smithi003.cldzke (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:38.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 -- 172.21.15.137:0/1364831182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 msgr2=0x7f9e480f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:38.578 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1364831182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e480f4be0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f9e3c009fb0 tx=0x7f9e3c009b30).stop 2023-11-15T05:59:38.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 -- 172.21.15.137:0/1364831182 shutdown_connections 2023-11-15T05:59:38.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1364831182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e480f4be0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:38.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 -- 172.21.15.137:0/1364831182 >> 172.21.15.137:0/1364831182 conn(0x7f9e480f02e0 msgr2=0x7f9e480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:38.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 -- 172.21.15.137:0/1364831182 shutdown_connections 2023-11-15T05:59:38.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.576+0000 7f9e4f116700 1 -- 172.21.15.137:0/1364831182 wait complete. 2023-11-15T05:59:38.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4f116700 1 Processor -- start 2023-11-15T05:59:38.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4f116700 1 -- start start 2023-11-15T05:59:38.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4f116700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:38.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4f116700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48106bf0 con 0x7f9e480f47c0 2023-11-15T05:59:38.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4ceb2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:38.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4ceb2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34436/0 (socket says 172.21.15.137:34436) 2023-11-15T05:59:38.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.577+0000 7f9e4ceb2700 1 -- 172.21.15.137:0/1610874883 learned_addr learned my addr 172.21.15.137:0/1610874883 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:38.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e4ceb2700 1 -- 172.21.15.137:0/1610874883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c005040 con 0x7f9e480f47c0 2023-11-15T05:59:38.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e4ceb2700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9e3c013510 tx=0x7f9e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:38.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3c009320 con 0x7f9e480f47c0 2023-11-15T05:59:38.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e3c013f00 con 0x7f9e480f47c0 2023-11-15T05:59:38.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e48106df0 con 0x7f9e480f47c0 2023-11-15T05:59:38.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e3c014d90 con 0x7f9e480f47c0 2023-11-15T05:59:38.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.578+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48107280 con 0x7f9e480f47c0 2023-11-15T05:59:38.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e3c0095f0 con 0x7f9e480f47c0 2023-11-15T05:59:38.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e48045ad0 con 0x7f9e480f47c0 2023-11-15T05:59:38.589 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e457fa700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:38.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e3c040fe0 con 0x7f9e480f47c0 2023-11-15T05:59:38.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e47fff700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:38.590 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.579+0000 7f9e47fff700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 0x7f9e38032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9e340096a0 tx=0x7f9e34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:38.592 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.583+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e3c01d7f0 con 0x7f9e480f47c0 2023-11-15T05:59:38.792 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.791+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e48108130 con 0x7f9e480f47c0 2023-11-15T05:59:38.793 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.791+0000 7f9e457fa700 1 -- 172.21.15.137:0/1610874883 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9e3c010230 con 0x7f9e480f47c0 2023-11-15T05:59:38.793 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:38.793 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:38.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 msgr2=0x7f9e38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:38.794 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 0x7f9e38032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9e340096a0 tx=0x7f9e34006b40).stop 2023-11-15T05:59:38.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 msgr2=0x7f9e48106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:38.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9e3c013510 tx=0x7f9e3c00b040).stop 2023-11-15T05:59:38.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 shutdown_connections 2023-11-15T05:59:38.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e3802ff60 0x7f9e38032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:38.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 --2- 172.21.15.137:0/1610874883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e480f47c0 0x7f9e48106700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 >> 172.21.15.137:0/1610874883 conn(0x7f9e480f02e0 msgr2=0x7f9e480f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 shutdown_connections 2023-11-15T05:59:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:38.793+0000 7f9e4f116700 1 -- 172.21.15.137:0/1610874883 wait complete. 2023-11-15T05:59:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:39.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:38.792645+0000 mon.smithi003 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.137:0/1610874883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:40.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:40.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:40.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:39.011185+0000 mgr.smithi003.cldzke (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:41.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 -- 172.21.15.137:0/3082509489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 msgr2=0x7f5cac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/3082509489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac0f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f5c9c009fb0 tx=0x7f5c9c009b30).stop 2023-11-15T05:59:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 -- 172.21.15.137:0/3082509489 shutdown_connections 2023-11-15T05:59:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/3082509489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac0f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 -- 172.21.15.137:0/3082509489 >> 172.21.15.137:0/3082509489 conn(0x7f5cac0f0b70 msgr2=0x7f5cac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 -- 172.21.15.137:0/3082509489 shutdown_connections 2023-11-15T05:59:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.412+0000 7f5cb3c73700 1 -- 172.21.15.137:0/3082509489 wait complete. 2023-11-15T05:59:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb3c73700 1 Processor -- start 2023-11-15T05:59:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb3c73700 1 -- start start 2023-11-15T05:59:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb3c73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:41.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb3c73700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cac105b20 con 0x7f5cac0f5410 2023-11-15T05:59:41.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb1a0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:41.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb1a0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37322/0 (socket says 172.21.15.137:37322) 2023-11-15T05:59:41.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.413+0000 7f5cb1a0f700 1 -- 172.21.15.137:0/2349092831 learned_addr learned my addr 172.21.15.137:0/2349092831 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:41.418 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5cb1a0f700 1 -- 172.21.15.137:0/2349092831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9c005040 con 0x7f5cac0f5410 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5cb1a0f700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f5c9c013510 tx=0x7f5c9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c9c009320 con 0x7f5cac0f5410 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac105d20 con 0x7f5cac0f5410 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c9c013f00 con 0x7f5cac0f5410 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c9c014d90 con 0x7f5cac0f5410 2023-11-15T05:59:41.419 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.414+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac1061b0 con 0x7f5cac0f5410 2023-11-15T05:59:41.420 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.415+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c9c014420 con 0x7f5cac0f5410 2023-11-15T05:59:41.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.415+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac045ad0 con 0x7f5cac0f5410 2023-11-15T05:59:41.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.415+0000 7f5ca27fc700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 0x7f5c98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:41.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.415+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c9c041070 con 0x7f5cac0f5410 2023-11-15T05:59:41.421 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.415+0000 7f5cb120e700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 0x7f5c98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:41.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.416+0000 7f5cb120e700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 0x7f5c98032470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ca80096a0 tx=0x7f5ca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:41.422 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.420+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c9c06d2b0 con 0x7f5cac0f5410 2023-11-15T05:59:41.628 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.627+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cac0fae60 con 0x7f5cac0f5410 2023-11-15T05:59:41.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.627+0000 7f5ca27fc700 1 -- 172.21.15.137:0/2349092831 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5c9c0095f0 con 0x7f5cac0f5410 2023-11-15T05:59:41.629 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:41.629 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:41.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 msgr2=0x7f5c98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:41.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 0x7f5c98032470 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ca80096a0 tx=0x7f5ca8006b40).stop 2023-11-15T05:59:41.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 msgr2=0x7f5cac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:41.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f5c9c013510 tx=0x7f5c9c00b040).stop 2023-11-15T05:59:41.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 shutdown_connections 2023-11-15T05:59:41.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5c9802ffb0 0x7f5c98032470 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 --2- 172.21.15.137:0/2349092831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5cac0f5410 0x7f5cac105630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 >> 172.21.15.137:0/2349092831 conn(0x7f5cac0f0b70 msgr2=0x7f5cac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 shutdown_connections 2023-11-15T05:59:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:41.629+0000 7f5cb3c73700 1 -- 172.21.15.137:0/2349092831 wait complete. 2023-11-15T05:59:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:41.011504+0000 mgr.smithi003.cldzke (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:42.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:41.628753+0000 mon.smithi003 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.137:0/2349092831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:43.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:43.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:43.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:43 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:43.011782+0000 mgr.smithi003.cldzke (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:44.229 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 -- 172.21.15.137:0/3478886977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 msgr2=0x7fef600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:44.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3478886977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef600f55c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fef50009fb0 tx=0x7fef50009b30).stop 2023-11-15T05:59:44.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 -- 172.21.15.137:0/3478886977 shutdown_connections 2023-11-15T05:59:44.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3478886977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef600f55c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:44.230 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 -- 172.21.15.137:0/3478886977 >> 172.21.15.137:0/3478886977 conn(0x7fef600f09a0 msgr2=0x7fef600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:44.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 -- 172.21.15.137:0/3478886977 shutdown_connections 2023-11-15T05:59:44.231 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 -- 172.21.15.137:0/3478886977 wait complete. 2023-11-15T05:59:44.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.228+0000 7fef64b6f700 1 Processor -- start 2023-11-15T05:59:44.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef64b6f700 1 -- start start 2023-11-15T05:59:44.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef64b6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:44.233 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef64b6f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef601058a0 con 0x7fef600f51a0 2023-11-15T05:59:44.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:44.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37330/0 (socket says 172.21.15.137:37330) 2023-11-15T05:59:44.234 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef5f7fe700 1 -- 172.21.15.137:0/3321558829 learned_addr learned my addr 172.21.15.137:0/3321558829 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:44.235 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.229+0000 7fef5f7fe700 1 -- 172.21.15.137:0/3321558829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50005040 con 0x7fef600f51a0 2023-11-15T05:59:44.235 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef5f7fe700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fef50013510 tx=0x7fef5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:44.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef50009320 con 0x7fef600f51a0 2023-11-15T05:59:44.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef60105aa0 con 0x7fef600f51a0 2023-11-15T05:59:44.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef50013f00 con 0x7fef600f51a0 2023-11-15T05:59:44.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef50014dc0 con 0x7fef600f51a0 2023-11-15T05:59:44.236 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef60105f30 con 0x7fef600f51a0 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.230+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef50014420 con 0x7fef600f51a0 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.231+0000 7fef47fff700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 0x7fef48032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.231+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef60045ad0 con 0x7fef600f51a0 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.231+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef50040ff0 con 0x7fef600f51a0 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.231+0000 7fef5effd700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 0x7fef48032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:44.237 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.231+0000 7fef5effd700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 0x7fef48032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fef540096a0 tx=0x7fef54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:44.238 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.235+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef5001db70 con 0x7fef600f51a0 2023-11-15T05:59:44.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.440+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef60106de0 con 0x7fef600f51a0 2023-11-15T05:59:44.442 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.441+0000 7fef47fff700 1 -- 172.21.15.137:0/3321558829 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fef5001c730 con 0x7fef600f51a0 2023-11-15T05:59:44.443 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:44.443 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:44.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 msgr2=0x7fef48032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:44.445 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 0x7fef48032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fef540096a0 tx=0x7fef54006b40).stop 2023-11-15T05:59:44.446 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 msgr2=0x7fef601053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:44.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fef50013510 tx=0x7fef5000b040).stop 2023-11-15T05:59:44.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 shutdown_connections 2023-11-15T05:59:44.447 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fef4802ffb0 0x7fef48032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:44.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 --2- 172.21.15.137:0/3321558829 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef600f51a0 0x7fef601053b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:44.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 >> 172.21.15.137:0/3321558829 conn(0x7fef600f09a0 msgr2=0x7fef600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:44.448 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 shutdown_connections 2023-11-15T05:59:44.449 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:44.442+0000 7fef64b6f700 1 -- 172.21.15.137:0/3321558829 wait complete. 2023-11-15T05:59:44.449 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:44.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:44.442285+0000 mon.smithi003 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.137:0/3321558829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:45.798 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:59:44.697714+0000 mgr.smithi003.cldzke (mgr.14164) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi003 ... 2023-11-15T05:59:45.798 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:59:45.798 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:59:45.799 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:55:23Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.799 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:56:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.799 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:56:38Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.799 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:57:52Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:58:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:59:07Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:59:45.801 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T05:59:45.801 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:59:45.801 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: main() 2023-11-15T05:59:45.801 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:59:45.801 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: r = ctx.func(ctx) 2023-11-15T05:59:45.802 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:59:45.802 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: return func(ctx) 2023-11-15T05:59:45.802 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:59:45.802 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:59:45.802 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:59:45.803 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:59:45.803 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:59:45.803 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: raise RuntimeError('uid/gid not found') 2023-11-15T05:59:45.803 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: RuntimeError: uid/gid not found 2023-11-15T05:59:45.803 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: yield (conn, connr) 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: code, '\n'.join(err))) 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi003 ... 2023-11-15T05:59:45.804 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T05:59:45.805 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:59:45.805 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:55:23Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.805 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:56:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.806 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:56:38Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.806 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:59:45.806 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T05:59:45.806 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T05:59:45.806 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:57:52Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.807 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:58:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.807 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr time="2023-11-15T05:59:07Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T05:59:45.807 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T05:59:45.807 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T05:59:45.807 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T05:59:45.808 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: main() 2023-11-15T05:59:45.808 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T05:59:45.808 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: r = ctx.func(ctx) 2023-11-15T05:59:45.808 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T05:59:45.808 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: return func(ctx) 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T05:59:45.809 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: raise RuntimeError('uid/gid not found') 2023-11-15T05:59:45.810 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: RuntimeError: uid/gid not found 2023-11-15T05:59:45.810 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:44.698984+0000 mon.smithi003 (mon.0) 228 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi003", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T05:59:45.810 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:44.701683+0000 mon.smithi003 (mon.0) 229 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi003", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T05:59:45.810 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:44.702574+0000 mon.smithi003 (mon.0) 230 : audit [DBG] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T05:59:45.810 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:59:44.703242+0000 mgr.smithi003.cldzke (mgr.14164) 157 : cephadm [INF] Deploying daemon crash.smithi003 on smithi003 2023-11-15T05:59:45.811 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:45.012103+0000 mgr.smithi003.cldzke (mgr.14164) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:45.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:45.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:46.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 -- 172.21.15.137:0/3466466975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 msgr2=0x7fcb1c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:46.994 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/3466466975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c0f8f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fcb18009fb0 tx=0x7fcb18009b30).stop 2023-11-15T05:59:46.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 -- 172.21.15.137:0/3466466975 shutdown_connections 2023-11-15T05:59:46.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/3466466975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c0f8f20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:46.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 -- 172.21.15.137:0/3466466975 >> 172.21.15.137:0/3466466975 conn(0x7fcb1c0f09a0 msgr2=0x7fcb1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:46.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 -- 172.21.15.137:0/3466466975 shutdown_connections 2023-11-15T05:59:46.995 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.991+0000 7fcb22fe1700 1 -- 172.21.15.137:0/3466466975 wait complete. 2023-11-15T05:59:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb22fe1700 1 Processor -- start 2023-11-15T05:59:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb22fe1700 1 -- start start 2023-11-15T05:59:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb22fe1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb22fe1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c1069f0 con 0x7fcb1c0601e0 2023-11-15T05:59:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb21fdf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:46.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb21fdf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37342/0 (socket says 172.21.15.137:37342) 2023-11-15T05:59:46.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb21fdf700 1 -- 172.21.15.137:0/2990430569 learned_addr learned my addr 172.21.15.137:0/2990430569 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:46.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb21fdf700 1 -- 172.21.15.137:0/2990430569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18005040 con 0x7fcb1c0601e0 2023-11-15T05:59:46.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.992+0000 7fcb21fdf700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcb18013510 tx=0x7fcb1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.993+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb18009320 con 0x7fcb1c0601e0 2023-11-15T05:59:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.993+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c106bf0 con 0x7fcb1c0601e0 2023-11-15T05:59:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.993+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c107080 con 0x7fcb1c0601e0 2023-11-15T05:59:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.993+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb18013f00 con 0x7fcb1c0601e0 2023-11-15T05:59:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.993+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb18014d60 con 0x7fcb1c0601e0 2023-11-15T05:59:47.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb180095f0 con 0x7fcb1c0601e0 2023-11-15T05:59:47.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb04004fa0 con 0x7fcb1c0601e0 2023-11-15T05:59:47.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb127fc700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:47.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb18040f60 con 0x7fcb1c0601e0 2023-11-15T05:59:47.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb217de700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:47.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.994+0000 7fcb217de700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 0x7fcb08032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fcb0c0096a0 tx=0x7fcb0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:47.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:46.999+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb1806d2b0 con 0x7fcb1c0601e0 2023-11-15T05:59:47.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.205+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb04005670 con 0x7fcb1c0601e0 2023-11-15T05:59:47.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.206+0000 7fcb127fc700 1 -- 172.21.15.137:0/2990430569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcb18014420 con 0x7fcb1c0601e0 2023-11-15T05:59:47.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:47.209 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:47.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 msgr2=0x7fcb08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:47.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 0x7fcb08032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fcb0c0096a0 tx=0x7fcb0c006b40).stop 2023-11-15T05:59:47.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 msgr2=0x7fcb1c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:47.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcb18013510 tx=0x7fcb1800b040).stop 2023-11-15T05:59:47.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 shutdown_connections 2023-11-15T05:59:47.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:47.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 --2- 172.21.15.137:0/2990430569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0601e0 0x7fcb1c106500 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:47.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 >> 172.21.15.137:0/2990430569 conn(0x7fcb1c0f09a0 msgr2=0x7fcb1c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:47.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 shutdown_connections 2023-11-15T05:59:47.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:47.208+0000 7fcb22fe1700 1 -- 172.21.15.137:0/2990430569 wait complete. 2023-11-15T05:59:47.212 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:47.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.331920+0000 mon.smithi003 (mon.0) 231 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:59:47.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.439319+0000 mon.smithi003 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:59:47.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.442744+0000 mon.smithi003 (mon.0) 233 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:59:47.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.443261+0000 mon.smithi003 (mon.0) 234 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:59:47.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.446963+0000 mon.smithi003 (mon.0) 235 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T05:59:47.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:47.207888+0000 mon.smithi003 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.137:0/2990430569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:48.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:48.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:48.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:46.443771+0000 mgr.smithi003.cldzke (mgr.14164) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T05:59:48.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T05:59:46.450586+0000 mgr.smithi003.cldzke (mgr.14164) 160 : cephadm [INF] Deploying daemon grafana.smithi003 on smithi003 2023-11-15T05:59:48.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:47.012518+0000 mgr.smithi003.cldzke (mgr.14164) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:49.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- 172.21.15.137:0/1443755969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 msgr2=0x7f8b0c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:49.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 --2- 172.21.15.137:0/1443755969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c0f66c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8afc009fb0 tx=0x7f8afc009b30).stop 2023-11-15T05:59:49.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- 172.21.15.137:0/1443755969 shutdown_connections 2023-11-15T05:59:49.837 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 --2- 172.21.15.137:0/1443755969 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c0f66c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:49.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- 172.21.15.137:0/1443755969 >> 172.21.15.137:0/1443755969 conn(0x7f8b0c0f1aa0 msgr2=0x7f8b0c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:49.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- 172.21.15.137:0/1443755969 shutdown_connections 2023-11-15T05:59:49.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- 172.21.15.137:0/1443755969 wait complete. 2023-11-15T05:59:49.838 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 Processor -- start 2023-11-15T05:59:49.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- start start 2023-11-15T05:59:49.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:49.839 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.835+0000 7f8b117a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c1069a0 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:49.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58018/0 (socket says 172.21.15.137:58018) 2023-11-15T05:59:49.841 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b0bfff700 1 -- 172.21.15.137:0/77148249 learned_addr learned my addr 172.21.15.137:0/77148249 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:49.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b0bfff700 1 -- 172.21.15.137:0/77148249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc005040 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b0bfff700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8afc013510 tx=0x7f8afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:49.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc009320 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8afc013f00 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c106ba0 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8afc014dc0 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.836+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c107030 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.837+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8afc0095f0 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.837+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c045ad0 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.837+0000 7f8b08ff9700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 0x7f8af4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:49.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.837+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8afc040f90 con 0x7f8b0c0f62a0 2023-11-15T05:59:49.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.838+0000 7f8b0b7fe700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 0x7f8af4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:49.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.838+0000 7f8b0b7fe700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 0x7f8af4032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8b000096a0 tx=0x7f8b00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:49.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:49.842+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8afc014420 con 0x7f8b0c0f62a0 2023-11-15T05:59:50.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.048+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b0c107e80 con 0x7f8b0c0f62a0 2023-11-15T05:59:50.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.050+0000 7f8b08ff9700 1 -- 172.21.15.137:0/77148249 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8afc010230 con 0x7f8b0c0f62a0 2023-11-15T05:59:50.051 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:50.051 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:50.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 msgr2=0x7f8af4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:50.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 0x7f8af4032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8b000096a0 tx=0x7f8b00006b40).stop 2023-11-15T05:59:50.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 msgr2=0x7f8b0c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:50.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8afc013510 tx=0x7f8afc00b040).stop 2023-11-15T05:59:50.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 shutdown_connections 2023-11-15T05:59:50.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f8af402ffb0 0x7f8af4032470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:50.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 --2- 172.21.15.137:0/77148249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b0c0f62a0 0x7f8b0c1064b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:50.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 >> 172.21.15.137:0/77148249 conn(0x7f8b0c0f1aa0 msgr2=0x7f8b0c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:50.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 shutdown_connections 2023-11-15T05:59:50.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:50.051+0000 7f8b117a9700 1 -- 172.21.15.137:0/77148249 wait complete. 2023-11-15T05:59:50.055 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:50.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:49.012851+0000 mgr.smithi003.cldzke (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:50.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:50.050547+0000 mon.smithi003 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.137:0/77148249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:51.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:51.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:51.013108+0000 mgr.smithi003.cldzke (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:52.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 -- 172.21.15.137:0/1955411671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 msgr2=0x7f84e80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:52.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/1955411671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e80f4be0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f84dc009fb0 tx=0x7f84dc009b30).stop 2023-11-15T05:59:52.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 -- 172.21.15.137:0/1955411671 shutdown_connections 2023-11-15T05:59:52.602 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/1955411671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e80f4be0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:52.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 -- 172.21.15.137:0/1955411671 >> 172.21.15.137:0/1955411671 conn(0x7f84e80f02e0 msgr2=0x7f84e80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:52.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 -- 172.21.15.137:0/1955411671 shutdown_connections 2023-11-15T05:59:52.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.599+0000 7f84ee8d7700 1 -- 172.21.15.137:0/1955411671 wait complete. 2023-11-15T05:59:52.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84ee8d7700 1 Processor -- start 2023-11-15T05:59:52.603 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84ee8d7700 1 -- start start 2023-11-15T05:59:52.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84ee8d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:52.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84ee8d7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8106bf0 con 0x7f84e80f47c0 2023-11-15T05:59:52.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84e7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:52.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84e7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58034/0 (socket says 172.21.15.137:58034) 2023-11-15T05:59:52.604 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.600+0000 7f84e7fff700 1 -- 172.21.15.137:0/2386732230 learned_addr learned my addr 172.21.15.137:0/2386732230 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:52.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84e7fff700 1 -- 172.21.15.137:0/2386732230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc005040 con 0x7f84e80f47c0 2023-11-15T05:59:52.606 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84e7fff700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f84dc014000 tx=0x7f84dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:52.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84dc0095f0 con 0x7f84e80f47c0 2023-11-15T05:59:52.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e8106df0 con 0x7f84e80f47c0 2023-11-15T05:59:52.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84dc007e70 con 0x7f84e80f47c0 2023-11-15T05:59:52.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84dc014ea0 con 0x7f84e80f47c0 2023-11-15T05:59:52.607 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.601+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8107280 con 0x7f84e80f47c0 2023-11-15T05:59:52.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84dc025070 con 0x7f84e80f47c0 2023-11-15T05:59:52.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e8045ad0 con 0x7f84e80f47c0 2023-11-15T05:59:52.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84e4ff9700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 0x7f84d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:52.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84dc0409b0 con 0x7f84e80f47c0 2023-11-15T05:59:52.608 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84e77fe700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 0x7f84d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:52.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.602+0000 7f84e77fe700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 0x7f84d0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f84d80096a0 tx=0x7f84d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:52.609 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.607+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84dc00f4e0 con 0x7f84e80f47c0 2023-11-15T05:59:52.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.814+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e8108130 con 0x7f84e80f47c0 2023-11-15T05:59:52.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.814+0000 7f84e4ff9700 1 -- 172.21.15.137:0/2386732230 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f84dc00f300 con 0x7f84e80f47c0 2023-11-15T05:59:52.816 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:52.817 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:52.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 msgr2=0x7f84d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:52.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 0x7f84d0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f84d80096a0 tx=0x7f84d8006b40).stop 2023-11-15T05:59:52.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 msgr2=0x7f84e8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:52.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f84dc014000 tx=0x7f84dc00b040).stop 2023-11-15T05:59:52.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 shutdown_connections 2023-11-15T05:59:52.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f84d002ff60 0x7f84d0032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:52.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 --2- 172.21.15.137:0/2386732230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84e80f47c0 0x7f84e8106700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:52.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 >> 172.21.15.137:0/2386732230 conn(0x7f84e80f02e0 msgr2=0x7f84e80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:52.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 shutdown_connections 2023-11-15T05:59:52.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:52.816+0000 7f84ee8d7700 1 -- 172.21.15.137:0/2386732230 wait complete. 2023-11-15T05:59:52.820 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:53.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:52.815847+0000 mon.smithi003 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.137:0/2386732230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:54.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:53.013429+0000 mgr.smithi003.cldzke (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:54.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:54.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:55.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 -- 172.21.15.137:0/1335540835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 msgr2=0x7fbf1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:55.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 --2- 172.21.15.137:0/1335540835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c0f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbf0c009fb0 tx=0x7fbf0c009b30).stop 2023-11-15T05:59:55.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 -- 172.21.15.137:0/1335540835 shutdown_connections 2023-11-15T05:59:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 --2- 172.21.15.137:0/1335540835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c0f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 -- 172.21.15.137:0/1335540835 >> 172.21.15.137:0/1335540835 conn(0x7fbf1c0f0b70 msgr2=0x7fbf1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 -- 172.21.15.137:0/1335540835 shutdown_connections 2023-11-15T05:59:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.364+0000 7fbf23986700 1 -- 172.21.15.137:0/1335540835 wait complete. 2023-11-15T05:59:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf23986700 1 Processor -- start 2023-11-15T05:59:55.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf23986700 1 -- start start 2023-11-15T05:59:55.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf23986700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:55.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf23986700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c060240 con 0x7fbf1c0f5410 2023-11-15T05:59:55.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf21722700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:55.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf21722700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58064/0 (socket says 172.21.15.137:58064) 2023-11-15T05:59:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf21722700 1 -- 172.21.15.137:0/3057132104 learned_addr learned my addr 172.21.15.137:0/3057132104 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.365+0000 7fbf21722700 1 -- 172.21.15.137:0/3057132104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c005040 con 0x7fbf1c0f5410 2023-11-15T05:59:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf21722700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fbf0c013510 tx=0x7fbf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0c009320 con 0x7fbf1c0f5410 2023-11-15T05:59:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf0c013f00 con 0x7fbf1c0f5410 2023-11-15T05:59:55.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c060440 con 0x7fbf1c0f5410 2023-11-15T05:59:55.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0c014d90 con 0x7fbf1c0f5410 2023-11-15T05:59:55.372 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.366+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c109ae0 con 0x7fbf1c0f5410 2023-11-15T05:59:55.373 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.367+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf0c014420 con 0x7fbf1c0f5410 2023-11-15T05:59:55.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.367+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c045ad0 con 0x7fbf1c0f5410 2023-11-15T05:59:55.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.367+0000 7fbf11ffb700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 0x7fbf08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:55.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.367+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf0c0410c0 con 0x7fbf1c0f5410 2023-11-15T05:59:55.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.367+0000 7fbf20f21700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 0x7fbf08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:55.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.368+0000 7fbf20f21700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 0x7fbf08032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fbf180096a0 tx=0x7fbf18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:55.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.371+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf0c0095f0 con 0x7fbf1c0f5410 2023-11-15T05:59:55.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.575+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf1c0fae60 con 0x7fbf1c0f5410 2023-11-15T05:59:55.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.576+0000 7fbf11ffb700 1 -- 172.21.15.137:0/3057132104 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbf0c01db70 con 0x7fbf1c0f5410 2023-11-15T05:59:55.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:55.578 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:55.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 msgr2=0x7fbf08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:55.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 0x7fbf08032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fbf180096a0 tx=0x7fbf18006b40).stop 2023-11-15T05:59:55.579 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 msgr2=0x7fbf1c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:55.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fbf0c013510 tx=0x7fbf0c00b040).stop 2023-11-15T05:59:55.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 shutdown_connections 2023-11-15T05:59:55.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbf0802ff60 0x7fbf08032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:55.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 --2- 172.21.15.137:0/3057132104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf1c0f5410 0x7fbf1c109430 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:55.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 >> 172.21.15.137:0/3057132104 conn(0x7fbf1c0f0b70 msgr2=0x7fbf1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:55.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 shutdown_connections 2023-11-15T05:59:55.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:55.577+0000 7fbf23986700 1 -- 172.21.15.137:0/3057132104 wait complete. 2023-11-15T05:59:55.585 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:56.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:55.013678+0000 mgr.smithi003.cldzke (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:55.576972+0000 mon.smithi003 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.137:0/3057132104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:56.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:56.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T05:59:58.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 -- 172.21.15.137:0/1886466222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 msgr2=0x7f14dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:58.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 --2- 172.21.15.137:0/1886466222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f14d0009fb0 tx=0x7f14d0009b30).stop 2023-11-15T05:59:58.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 -- 172.21.15.137:0/1886466222 shutdown_connections 2023-11-15T05:59:58.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 --2- 172.21.15.137:0/1886466222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:58.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 -- 172.21.15.137:0/1886466222 >> 172.21.15.137:0/1886466222 conn(0x7f14dc0f1c70 msgr2=0x7f14dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:58.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 -- 172.21.15.137:0/1886466222 shutdown_connections 2023-11-15T05:59:58.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.098+0000 7f14e12e8700 1 -- 172.21.15.137:0/1886466222 wait complete. 2023-11-15T05:59:58.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14e12e8700 1 Processor -- start 2023-11-15T05:59:58.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14e12e8700 1 -- start start 2023-11-15T05:59:58.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14e12e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:58.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14e12e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc106c20 con 0x7f14dc0f6510 2023-11-15T05:59:58.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14daffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:58.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14daffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58086/0 (socket says 172.21.15.137:58086) 2023-11-15T05:59:58.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.099+0000 7f14daffd700 1 -- 172.21.15.137:0/2041666714 learned_addr learned my addr 172.21.15.137:0/2041666714 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T05:59:58.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14daffd700 1 -- 172.21.15.137:0/2041666714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0005040 con 0x7f14dc0f6510 2023-11-15T05:59:58.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14daffd700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:58.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0009320 con 0x7f14dc0f6510 2023-11-15T05:59:58.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d0013f00 con 0x7f14dc0f6510 2023-11-15T05:59:58.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0014d90 con 0x7f14dc0f6510 2023-11-15T05:59:58.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc106e20 con 0x7f14dc0f6510 2023-11-15T05:59:58.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.100+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc1072b0 con 0x7f14dc0f6510 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.101+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14d00095f0 con 0x7f14dc0f6510 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.101+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14dc045ad0 con 0x7f14dc0f6510 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.101+0000 7f14c37fe700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 0x7f14c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.101+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14d00410c0 con 0x7f14dc0f6510 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.101+0000 7f14da7fc700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 0x7f14c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T05:59:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.102+0000 7f14da7fc700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 0x7f14c4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f14cc0096a0 tx=0x7f14cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T05:59:58.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.106+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14d00263a0 con 0x7f14dc0f6510 2023-11-15T05:59:58.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.313+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14dc0fbf60 con 0x7f14dc0f6510 2023-11-15T05:59:58.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.313+0000 7f14c37fe700 1 -- 172.21.15.137:0/2041666714 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f14d0010230 con 0x7f14dc0f6510 2023-11-15T05:59:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T05:59:58.315 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T05:59:58.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 msgr2=0x7f14c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:58.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 0x7f14c4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f14cc0096a0 tx=0x7f14cc006b40).stop 2023-11-15T05:59:58.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 msgr2=0x7f14dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T05:59:58.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).stop 2023-11-15T05:59:58.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 shutdown_connections 2023-11-15T05:59:58.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14c402ff60 0x7f14c4032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:58.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 --2- 172.21.15.137:0/2041666714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14dc0f6510 0x7f14dc106730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T05:59:58.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 >> 172.21.15.137:0/2041666714 conn(0x7f14dc0f1c70 msgr2=0x7f14dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T05:59:58.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 shutdown_connections 2023-11-15T05:59:58.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T05:59:58.315+0000 7f14e12e8700 1 -- 172.21.15.137:0/2041666714 wait complete. 2023-11-15T05:59:58.319 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T05:59:58.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:57.013928+0000 mgr.smithi003.cldzke (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T05:59:59.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 05:59:59 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T05:59:58.314646+0000 mon.smithi003 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.137:0/2041666714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T05:59:59.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T05:59:59.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:00.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T05:59:59.014187+0000 mgr.smithi003.cldzke (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:00.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- 172.21.15.137:0/2050711393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 msgr2=0x7f21f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:00.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/2050711393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f40f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f21f0009fb0 tx=0x7f21f0009b30).stop 2023-11-15T06:00:00.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- 172.21.15.137:0/2050711393 shutdown_connections 2023-11-15T06:00:00.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/2050711393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f40f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:00.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- 172.21.15.137:0/2050711393 >> 172.21.15.137:0/2050711393 conn(0x7f21f40f0b70 msgr2=0x7f21f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:00.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- 172.21.15.137:0/2050711393 shutdown_connections 2023-11-15T06:00:00.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- 172.21.15.137:0/2050711393 wait complete. 2023-11-15T06:00:00.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 Processor -- start 2023-11-15T06:00:00.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- start start 2023-11-15T06:00:00.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:00.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.908+0000 7f21fb3ee700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f4106c10 con 0x7f21f40f5410 2023-11-15T06:00:00.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21f918a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:00.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21f918a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:35734/0 (socket says 172.21.15.137:35734) 2023-11-15T06:00:00.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21f918a700 1 -- 172.21.15.137:0/651027879 learned_addr learned my addr 172.21.15.137:0/651027879 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:00.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21f918a700 1 -- 172.21.15.137:0/651027879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0005040 con 0x7f21f40f5410 2023-11-15T06:00:00.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21f918a700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f21f0013510 tx=0x7f21f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:00.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.909+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f0009320 con 0x7f21f40f5410 2023-11-15T06:00:00.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.910+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f4106e10 con 0x7f21f40f5410 2023-11-15T06:00:00.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.910+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21f0013f00 con 0x7f21f40f5410 2023-11-15T06:00:00.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.910+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21f0014d90 con 0x7f21f40f5410 2023-11-15T06:00:00.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.910+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f41072a0 con 0x7f21f40f5410 2023-11-15T06:00:00.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21f00095f0 con 0x7f21f40f5410 2023-11-15T06:00:00.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f4045ad0 con 0x7f21f40f5410 2023-11-15T06:00:00.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21e9ffb700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:00.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21f0040fc0 con 0x7f21f40f5410 2023-11-15T06:00:00.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21f8989700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:00.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.911+0000 7f21f8989700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 0x7f21e0032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f21e40096a0 tx=0x7f21e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:00.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:00.916+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21f0014420 con 0x7f21f40f5410 2023-11-15T06:00:01.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.121+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21f40fae60 con 0x7f21f40f5410 2023-11-15T06:00:01.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.121+0000 7f21e9ffb700 1 -- 172.21.15.137:0/651027879 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f21f0010230 con 0x7f21f40f5410 2023-11-15T06:00:01.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:01.123 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:01.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 msgr2=0x7f21e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:01.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 0x7f21e0032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f21e40096a0 tx=0x7f21e4006b40).stop 2023-11-15T06:00:01.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 msgr2=0x7f21f4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:01.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f21f0013510 tx=0x7f21f000b040).stop 2023-11-15T06:00:01.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 shutdown_connections 2023-11-15T06:00:01.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:01.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 --2- 172.21.15.137:0/651027879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f21f40f5410 0x7f21f4106720 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:01.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 >> 172.21.15.137:0/651027879 conn(0x7f21f40f0b70 msgr2=0x7f21f405fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:01.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 shutdown_connections 2023-11-15T06:00:01.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:01.123+0000 7f21fb3ee700 1 -- 172.21.15.137:0/651027879 wait complete. 2023-11-15T06:00:01.127 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:01.860 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:01.014476+0000 mgr.smithi003.cldzke (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:01.860 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:01.122737+0000 mon.smithi003 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.137:0/651027879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:02.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:02.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:03.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 -- 172.21.15.137:0/852203434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 msgr2=0x7fda8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:03.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 --2- 172.21.15.137:0/852203434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c0f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fda80009fb0 tx=0x7fda80009b30).stop 2023-11-15T06:00:03.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 -- 172.21.15.137:0/852203434 shutdown_connections 2023-11-15T06:00:03.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 --2- 172.21.15.137:0/852203434 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c0f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:03.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 -- 172.21.15.137:0/852203434 >> 172.21.15.137:0/852203434 conn(0x7fda8c0f1c70 msgr2=0x7fda8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:03.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 -- 172.21.15.137:0/852203434 shutdown_connections 2023-11-15T06:00:03.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 -- 172.21.15.137:0/852203434 wait complete. 2023-11-15T06:00:03.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.687+0000 7fda91bb4700 1 Processor -- start 2023-11-15T06:00:03.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda91bb4700 1 -- start start 2023-11-15T06:00:03.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda91bb4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:03.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda91bb4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c069e20 con 0x7fda8c0f6510 2023-11-15T06:00:03.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda8b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:03.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda8b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:35744/0 (socket says 172.21.15.137:35744) 2023-11-15T06:00:03.693 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda8b7fe700 1 -- 172.21.15.137:0/1647614988 learned_addr learned my addr 172.21.15.137:0/1647614988 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:03.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda8b7fe700 1 -- 172.21.15.137:0/1647614988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80005040 con 0x7fda8c0f6510 2023-11-15T06:00:03.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.688+0000 7fda8b7fe700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fda80013510 tx=0x7fda8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.689+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda80009320 con 0x7fda8c0f6510 2023-11-15T06:00:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.689+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c069fc0 con 0x7fda8c0f6510 2023-11-15T06:00:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.689+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda80013f00 con 0x7fda8c0f6510 2023-11-15T06:00:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.689+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c06a450 con 0x7fda8c0f6510 2023-11-15T06:00:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.689+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda80014d90 con 0x7fda8c0f6510 2023-11-15T06:00:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda800263a0 con 0x7fda8c0f6510 2023-11-15T06:00:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda73fff700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 0x7fda74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda80041f20 con 0x7fda8c0f6510 2023-11-15T06:00:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda8affd700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 0x7fda74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c059100 con 0x7fda8c0f6510 2023-11-15T06:00:03.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.690+0000 7fda8affd700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 0x7fda74032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fda7c0096a0 tx=0x7fda7c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:03.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.698+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda80012920 con 0x7fda8c0f6510 2023-11-15T06:00:03.906 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.906+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda8c045ad0 con 0x7fda8c0f6510 2023-11-15T06:00:03.907 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.906+0000 7fda73fff700 1 -- 172.21.15.137:0/1647614988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fda8002b020 con 0x7fda8c0f6510 2023-11-15T06:00:03.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:03.907 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:03.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 msgr2=0x7fda74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:03.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 0x7fda74032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fda7c0096a0 tx=0x7fda7c006b40).stop 2023-11-15T06:00:03.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 msgr2=0x7fda8c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:03.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fda80013510 tx=0x7fda8000b040).stop 2023-11-15T06:00:03.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 shutdown_connections 2023-11-15T06:00:03.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fda7402ff60 0x7fda74032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 --2- 172.21.15.137:0/1647614988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda8c0f6510 0x7fda8c06b9a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 >> 172.21.15.137:0/1647614988 conn(0x7fda8c0f1c70 msgr2=0x7fda8c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 shutdown_connections 2023-11-15T06:00:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:03.908+0000 7fda91bb4700 1 -- 172.21.15.137:0/1647614988 wait complete. 2023-11-15T06:00:03.912 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:04.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:03.014850+0000 mgr.smithi003.cldzke (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:04.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:03.238744+0000 mon.smithi003 (mon.0) 242 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' 2023-11-15T06:00:04.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T06:00:03.240272+0000 mgr.smithi003.cldzke (mgr.14164) 170 : cephadm [INF] Deploying daemon node-exporter.smithi003 on smithi003 2023-11-15T06:00:04.682 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:03.906837+0000 mon.smithi003 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.137:0/1647614988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:05.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:05.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:06.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- 172.21.15.137:0/300580498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 msgr2=0x7f5dc00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/300580498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc00f66c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f5dbc009fb0 tx=0x7f5dbc009b30).stop 2023-11-15T06:00:06.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- 172.21.15.137:0/300580498 shutdown_connections 2023-11-15T06:00:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/300580498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc00f66c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- 172.21.15.137:0/300580498 >> 172.21.15.137:0/300580498 conn(0x7f5dc00f1aa0 msgr2=0x7f5dc00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:06.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- 172.21.15.137:0/300580498 shutdown_connections 2023-11-15T06:00:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- 172.21.15.137:0/300580498 wait complete. 2023-11-15T06:00:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 Processor -- start 2023-11-15T06:00:06.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- start start 2023-11-15T06:00:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.501+0000 7f5dc64ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0069e20 con 0x7f5dc00f62a0 2023-11-15T06:00:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc54ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:06.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc54ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:35762/0 (socket says 172.21.15.137:35762) 2023-11-15T06:00:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc54ac700 1 -- 172.21.15.137:0/2703267298 learned_addr learned my addr 172.21.15.137:0/2703267298 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc54ac700 1 -- 172.21.15.137:0/2703267298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc005040 con 0x7f5dc00f62a0 2023-11-15T06:00:06.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc54ac700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5dbc013510 tx=0x7f5dbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dbc009320 con 0x7f5dc00f62a0 2023-11-15T06:00:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc006a020 con 0x7f5dc00f62a0 2023-11-15T06:00:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dbc013f00 con 0x7f5dc00f62a0 2023-11-15T06:00:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.502+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dbc014dc0 con 0x7f5dc00f62a0 2023-11-15T06:00:06.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.503+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc006a4b0 con 0x7f5dc00f62a0 2023-11-15T06:00:06.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.503+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5dbc0095f0 con 0x7f5dc00f62a0 2023-11-15T06:00:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.503+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0059100 con 0x7f5dc00f62a0 2023-11-15T06:00:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.503+0000 7f5db5ffb700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 0x7f5dac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.504+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5dbc041100 con 0x7f5dc00f62a0 2023-11-15T06:00:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.504+0000 7f5dc4cab700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 0x7f5dac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.504+0000 7f5dc4cab700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 0x7f5dac032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5db00096a0 tx=0x7f5db0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:06.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.508+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5dbc014420 con 0x7f5dc00f62a0 2023-11-15T06:00:06.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:05.015178+0000 mgr.smithi003.cldzke (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:06.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.712+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5dc00fbce0 con 0x7f5dc00f62a0 2023-11-15T06:00:06.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.712+0000 7f5db5ffb700 1 -- 172.21.15.137:0/2703267298 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5dbc010230 con 0x7f5dc00f62a0 2023-11-15T06:00:06.714 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:06.714 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:06.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 msgr2=0x7f5dac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:06.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 0x7f5dac032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5db00096a0 tx=0x7f5db0006b40).stop 2023-11-15T06:00:06.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 msgr2=0x7f5dc006b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5dbc013510 tx=0x7f5dbc00b040).stop 2023-11-15T06:00:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 shutdown_connections 2023-11-15T06:00:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5dac02ff60 0x7f5dac032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 --2- 172.21.15.137:0/2703267298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5dc00f62a0 0x7f5dc006b930 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 >> 172.21.15.137:0/2703267298 conn(0x7f5dc00f1aa0 msgr2=0x7f5dc00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 shutdown_connections 2023-11-15T06:00:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:06.714+0000 7f5dc64ae700 1 -- 172.21.15.137:0/2703267298 wait complete. 2023-11-15T06:00:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:07.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:07 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:06.712941+0000 mon.smithi003 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.137:0/2703267298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:08.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:08.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:08.680 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:07.015516+0000 mgr.smithi003.cldzke (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:09.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- 172.21.15.137:0/4144704994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 msgr2=0x7f96a80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:09.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 --2- 172.21.15.137:0/4144704994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f7d00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9698009fb0 tx=0x7f9698009b30).stop 2023-11-15T06:00:09.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- 172.21.15.137:0/4144704994 shutdown_connections 2023-11-15T06:00:09.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 --2- 172.21.15.137:0/4144704994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f7d00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:09.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- 172.21.15.137:0/4144704994 >> 172.21.15.137:0/4144704994 conn(0x7f96a80f0120 msgr2=0x7f96a80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:09.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- 172.21.15.137:0/4144704994 shutdown_connections 2023-11-15T06:00:09.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- 172.21.15.137:0/4144704994 wait complete. 2023-11-15T06:00:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 Processor -- start 2023-11-15T06:00:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- start start 2023-11-15T06:00:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.289+0000 7f96a7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a80f5630 con 0x7f96a80f78e0 2023-11-15T06:00:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:59250/0 (socket says 172.21.15.137:59250) 2023-11-15T06:00:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a6ffd700 1 -- 172.21.15.137:0/2850718818 learned_addr learned my addr 172.21.15.137:0/2850718818 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a6ffd700 1 -- 172.21.15.137:0/2850718818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698005040 con 0x7f96a80f78e0 2023-11-15T06:00:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a6ffd700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9698013510 tx=0x7f969800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9698009320 con 0x7f96a80f78e0 2023-11-15T06:00:09.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a80f57d0 con 0x7f96a80f78e0 2023-11-15T06:00:09.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.290+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9698013f00 con 0x7f96a80f78e0 2023-11-15T06:00:09.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.291+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9698014dc0 con 0x7f96a80f78e0 2023-11-15T06:00:09.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.291+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a80f5c60 con 0x7f96a80f78e0 2023-11-15T06:00:09.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.291+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96980095f0 con 0x7f96a80f78e0 2023-11-15T06:00:09.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.291+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a8059100 con 0x7f96a80f78e0 2023-11-15T06:00:09.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.292+0000 7f968f7fe700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 0x7f9690032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:09.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.292+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9698041000 con 0x7f96a80f78e0 2023-11-15T06:00:09.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.292+0000 7f96a67fc700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 0x7f9690032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:09.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.292+0000 7f96a67fc700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 0x7f9690032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f969c0096a0 tx=0x7f969c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:09.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.296+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9698014420 con 0x7f96a80f78e0 2023-11-15T06:00:09.503 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.502+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96a80fbce0 con 0x7f96a80f78e0 2023-11-15T06:00:09.504 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.503+0000 7f968f7fe700 1 -- 172.21.15.137:0/2850718818 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9698010230 con 0x7f96a80f78e0 2023-11-15T06:00:09.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:09.504 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:09.505 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 msgr2=0x7f9690032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:09.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 0x7f9690032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f969c0096a0 tx=0x7f969c006b40).stop 2023-11-15T06:00:09.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 msgr2=0x7f96a80f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:09.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9698013510 tx=0x7f969800b040).stop 2023-11-15T06:00:09.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 shutdown_connections 2023-11-15T06:00:09.506 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f969002ff60 0x7f9690032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:09.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 --2- 172.21.15.137:0/2850718818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a80f78e0 0x7f96a80f71b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:09.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 >> 172.21.15.137:0/2850718818 conn(0x7f96a80f0120 msgr2=0x7f96a80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:09.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 shutdown_connections 2023-11-15T06:00:09.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:09.504+0000 7f96a7fff700 1 -- 172.21.15.137:0/2850718818 wait complete. 2023-11-15T06:00:09.507 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:10.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:09.015882+0000 mgr.smithi003.cldzke (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:10.681 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:09.503317+0000 mon.smithi003 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.137:0/2850718818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:10.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:10.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:11.016252+0000 mgr.smithi003.cldzke (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:12.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- 172.21.15.137:0/1241790478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 msgr2=0x7f048c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:12.105 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 --2- 172.21.15.137:0/1241790478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c0f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f0480009fb0 tx=0x7f0480009b30).stop 2023-11-15T06:00:12.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- 172.21.15.137:0/1241790478 shutdown_connections 2023-11-15T06:00:12.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 --2- 172.21.15.137:0/1241790478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c0f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:12.106 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- 172.21.15.137:0/1241790478 >> 172.21.15.137:0/1241790478 conn(0x7f048c0f1c70 msgr2=0x7f048c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:12.107 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- 172.21.15.137:0/1241790478 shutdown_connections 2023-11-15T06:00:12.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- 172.21.15.137:0/1241790478 wait complete. 2023-11-15T06:00:12.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 Processor -- start 2023-11-15T06:00:12.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.101+0000 7f04909a0700 1 -- start start 2023-11-15T06:00:12.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f04909a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:12.108 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f04909a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c106bc0 con 0x7f048c0f6510 2023-11-15T06:00:12.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f048a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:12.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f048a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:59268/0 (socket says 172.21.15.137:59268) 2023-11-15T06:00:12.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f048a59c700 1 -- 172.21.15.137:0/3542145025 learned_addr learned my addr 172.21.15.137:0/3542145025 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:12.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f048a59c700 1 -- 172.21.15.137:0/3542145025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0480005040 con 0x7f048c0f6510 2023-11-15T06:00:12.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f048a59c700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f0480013510 tx=0x7f048000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:12.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0480009320 con 0x7f048c0f6510 2023-11-15T06:00:12.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c106dc0 con 0x7f048c0f6510 2023-11-15T06:00:12.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0480013f00 con 0x7f048c0f6510 2023-11-15T06:00:12.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0480014d90 con 0x7f048c0f6510 2023-11-15T06:00:12.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.102+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c107250 con 0x7f048c0f6510 2023-11-15T06:00:12.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.103+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04800095f0 con 0x7f048c0f6510 2023-11-15T06:00:12.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.103+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c045ad0 con 0x7f048c0f6510 2023-11-15T06:00:12.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.103+0000 7f047affd700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 0x7f0474032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:12.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.103+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0480041080 con 0x7f048c0f6510 2023-11-15T06:00:12.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.104+0000 7f0489d9b700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 0x7f0474032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:12.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.104+0000 7f0489d9b700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 0x7f0474032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f047c0096a0 tx=0x7f047c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:12.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.108+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0480014420 con 0x7f048c0f6510 2023-11-15T06:00:12.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.320+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f048c108100 con 0x7f048c0f6510 2023-11-15T06:00:12.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.321+0000 7f047affd700 1 -- 172.21.15.137:0/3542145025 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f048001d7f0 con 0x7f048c0f6510 2023-11-15T06:00:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:12.322 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:12.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 msgr2=0x7f0474032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:12.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 0x7f0474032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f047c0096a0 tx=0x7f047c006b40).stop 2023-11-15T06:00:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 msgr2=0x7f048c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f0480013510 tx=0x7f048000b040).stop 2023-11-15T06:00:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 shutdown_connections 2023-11-15T06:00:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f047402ff60 0x7f0474032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 --2- 172.21.15.137:0/3542145025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f048c0f6510 0x7f048c1066d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:12.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 >> 172.21.15.137:0/3542145025 conn(0x7f048c0f1c70 msgr2=0x7f048c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:12.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 shutdown_connections 2023-11-15T06:00:12.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:12.322+0000 7f04909a0700 1 -- 172.21.15.137:0/3542145025 wait complete. 2023-11-15T06:00:12.326 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:12.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:12.321289+0000 mon.smithi003 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.137:0/3542145025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:13.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:13.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:13.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:13.016606+0000 mgr.smithi003.cldzke (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:14.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.910+0000 7f2abb901700 1 -- 172.21.15.137:0/3432683772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 msgr2=0x7f2ab40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:14.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.910+0000 7f2abb901700 1 --2- 172.21.15.137:0/3432683772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab40f90f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f2aa4009fb0 tx=0x7f2aa4009b30).stop 2023-11-15T06:00:14.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 -- 172.21.15.137:0/3432683772 shutdown_connections 2023-11-15T06:00:14.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 --2- 172.21.15.137:0/3432683772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab40f90f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:14.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 -- 172.21.15.137:0/3432683772 >> 172.21.15.137:0/3432683772 conn(0x7f2ab40f0b70 msgr2=0x7f2ab40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:14.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 -- 172.21.15.137:0/3432683772 shutdown_connections 2023-11-15T06:00:14.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 -- 172.21.15.137:0/3432683772 wait complete. 2023-11-15T06:00:14.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 Processor -- start 2023-11-15T06:00:14.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.911+0000 7f2abb901700 1 -- start start 2023-11-15T06:00:14.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2abb901700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:14.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2abb901700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab4106bd0 con 0x7f2ab4060280 2023-11-15T06:00:14.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2ab969d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:14.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2ab969d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:59286/0 (socket says 172.21.15.137:59286) 2023-11-15T06:00:14.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2ab969d700 1 -- 172.21.15.137:0/481172569 learned_addr learned my addr 172.21.15.137:0/481172569 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:14.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2ab969d700 1 -- 172.21.15.137:0/481172569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4005040 con 0x7f2ab4060280 2023-11-15T06:00:14.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2ab969d700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2aa4013510 tx=0x7f2aa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:14.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa4009320 con 0x7f2ab4060280 2023-11-15T06:00:14.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab4106dd0 con 0x7f2ab4060280 2023-11-15T06:00:14.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa4013f00 con 0x7f2ab4060280 2023-11-15T06:00:14.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.912+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa4014dc0 con 0x7f2ab4060280 2023-11-15T06:00:14.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.913+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab4107260 con 0x7f2ab4060280 2023-11-15T06:00:14.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.913+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2aa40095f0 con 0x7f2ab4060280 2023-11-15T06:00:14.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.913+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4045ad0 con 0x7f2ab4060280 2023-11-15T06:00:14.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.914+0000 7f2aa9ffb700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 0x7f2aa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:14.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.914+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2aa4041020 con 0x7f2ab4060280 2023-11-15T06:00:14.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.914+0000 7f2ab8e9c700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 0x7f2aa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:14.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.914+0000 7f2ab8e9c700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 0x7f2aa0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2ab00096a0 tx=0x7f2ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:14.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:14.918+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2aa4014420 con 0x7f2ab4060280 2023-11-15T06:00:15.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.121+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ab4108040 con 0x7f2ab4060280 2023-11-15T06:00:15.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.122+0000 7f2aa9ffb700 1 -- 172.21.15.137:0/481172569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2aa401c920 con 0x7f2ab4060280 2023-11-15T06:00:15.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:15.123 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:15.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 msgr2=0x7f2aa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:15.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 0x7f2aa0032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2ab00096a0 tx=0x7f2ab0006b40).stop 2023-11-15T06:00:15.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 msgr2=0x7f2ab41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:15.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2aa4013510 tx=0x7f2aa400b040).stop 2023-11-15T06:00:15.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 shutdown_connections 2023-11-15T06:00:15.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2aa002ff60 0x7f2aa0032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:15.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 --2- 172.21.15.137:0/481172569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ab4060280 0x7f2ab41066e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:15.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 >> 172.21.15.137:0/481172569 conn(0x7f2ab40f0b70 msgr2=0x7f2ab40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:15.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 shutdown_connections 2023-11-15T06:00:15.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:15.123+0000 7f2abb901700 1 -- 172.21.15.137:0/481172569 wait complete. 2023-11-15T06:00:15.126 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:15.016916+0000 mgr.smithi003.cldzke (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:15.122084+0000 mon.smithi003 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.137:0/481172569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:16.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:16.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:17.639 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 -- 172.21.15.137:0/2967957008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 msgr2=0x7f48440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:17.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 --2- 172.21.15.137:0/2967957008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f48440f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4834009fb0 tx=0x7f4834009b30).stop 2023-11-15T06:00:17.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 -- 172.21.15.137:0/2967957008 shutdown_connections 2023-11-15T06:00:17.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 --2- 172.21.15.137:0/2967957008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f48440f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:17.640 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 -- 172.21.15.137:0/2967957008 >> 172.21.15.137:0/2967957008 conn(0x7f48440f0b70 msgr2=0x7f48440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:17.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 -- 172.21.15.137:0/2967957008 shutdown_connections 2023-11-15T06:00:17.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.637+0000 7f484b8e2700 1 -- 172.21.15.137:0/2967957008 wait complete. 2023-11-15T06:00:17.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484b8e2700 1 Processor -- start 2023-11-15T06:00:17.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484b8e2700 1 -- start start 2023-11-15T06:00:17.641 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484b8e2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:17.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484b8e2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4844060240 con 0x7f48440f5410 2023-11-15T06:00:17.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484967e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:17.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484967e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:59308/0 (socket says 172.21.15.137:59308) 2023-11-15T06:00:17.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484967e700 1 -- 172.21.15.137:0/4250274446 learned_addr learned my addr 172.21.15.137:0/4250274446 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:17.642 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484967e700 1 -- 172.21.15.137:0/4250274446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4834005040 con 0x7f48440f5410 2023-11-15T06:00:17.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.638+0000 7f484967e700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f4834013510 tx=0x7f483400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:17.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.639+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4834009320 con 0x7f48440f5410 2023-11-15T06:00:17.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.639+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4844060440 con 0x7f48440f5410 2023-11-15T06:00:17.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.639+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4834013f00 con 0x7f48440f5410 2023-11-15T06:00:17.643 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.639+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4834014d90 con 0x7f48440f5410 2023-11-15T06:00:17.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.639+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4844109ae0 con 0x7f48440f5410 2023-11-15T06:00:17.644 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48340263a0 con 0x7f48440f5410 2023-11-15T06:00:17.645 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f4839ffb700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 0x7f4830032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:17.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4834041060 con 0x7f48440f5410 2023-11-15T06:00:17.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4844045ad0 con 0x7f48440f5410 2023-11-15T06:00:17.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f4848e7d700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 0x7f4830032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:17.646 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.640+0000 7f4848e7d700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 0x7f4830032470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f48400096a0 tx=0x7f4840006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:17.647 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.645+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4834012920 con 0x7f48440f5410 2023-11-15T06:00:17.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.854+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48440faf50 con 0x7f48440f5410 2023-11-15T06:00:17.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.854+0000 7f4839ffb700 1 -- 172.21.15.137:0/4250274446 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f483402b020 con 0x7f48440f5410 2023-11-15T06:00:17.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:17.856 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:17.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 msgr2=0x7f4830032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:17.858 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 0x7f4830032470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f48400096a0 tx=0x7f4840006b40).stop 2023-11-15T06:00:17.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 msgr2=0x7f4844109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:17.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f4834013510 tx=0x7f483400b040).stop 2023-11-15T06:00:17.859 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 shutdown_connections 2023-11-15T06:00:17.860 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f483002ffb0 0x7f4830032470 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:17.860 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 --2- 172.21.15.137:0/4250274446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48440f5410 0x7f4844109430 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:17.860 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 >> 172.21.15.137:0/4250274446 conn(0x7f48440f0b70 msgr2=0x7f48440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:17.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 shutdown_connections 2023-11-15T06:00:17.868 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:17.856+0000 7f484b8e2700 1 -- 172.21.15.137:0/4250274446 wait complete. 2023-11-15T06:00:17.868 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:17.017276+0000 mgr.smithi003.cldzke (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:17.854870+0000 mon.smithi003 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.137:0/4250274446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:19.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:19.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:20.345 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 -- 172.21.15.137:0/3807481556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 msgr2=0x7f58080f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:20.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 --2- 172.21.15.137:0/3807481556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58080f9190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f57fc009fb0 tx=0x7f57fc009b30).stop 2023-11-15T06:00:20.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 -- 172.21.15.137:0/3807481556 shutdown_connections 2023-11-15T06:00:20.346 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 --2- 172.21.15.137:0/3807481556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58080f9190 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 -- 172.21.15.137:0/3807481556 >> 172.21.15.137:0/3807481556 conn(0x7f58080f0b70 msgr2=0x7f58080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 -- 172.21.15.137:0/3807481556 shutdown_connections 2023-11-15T06:00:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.344+0000 7f580e1af700 1 -- 172.21.15.137:0/3807481556 wait complete. 2023-11-15T06:00:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f580e1af700 1 Processor -- start 2023-11-15T06:00:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f580e1af700 1 -- start start 2023-11-15T06:00:20.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f580e1af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:20.348 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f580e1af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5808106bd0 con 0x7f58080601e0 2023-11-15T06:00:20.349 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f58077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:20.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f58077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37006/0 (socket says 172.21.15.137:37006) 2023-11-15T06:00:20.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.345+0000 7f58077fe700 1 -- 172.21.15.137:0/1253677758 learned_addr learned my addr 172.21.15.137:0/1253677758 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:20.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f58077fe700 1 -- 172.21.15.137:0/1253677758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc005040 con 0x7f58080601e0 2023-11-15T06:00:20.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f58077fe700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f57fc004dd0 tx=0x7f57fc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:20.350 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57fc018070 con 0x7f58080601e0 2023-11-15T06:00:20.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5808106dd0 con 0x7f58080601e0 2023-11-15T06:00:20.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57fc013410 con 0x7f58080601e0 2023-11-15T06:00:20.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57fc01c3c0 con 0x7f58080601e0 2023-11-15T06:00:20.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.346+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5808107260 con 0x7f58080601e0 2023-11-15T06:00:20.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.347+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57fc01c630 con 0x7f58080601e0 2023-11-15T06:00:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.347+0000 7f57effff700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 0x7f57f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.347+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5808045ad0 con 0x7f58080601e0 2023-11-15T06:00:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.347+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57fc041450 con 0x7f58080601e0 2023-11-15T06:00:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.347+0000 7f5806ffd700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 0x7f57f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.348+0000 7f5806ffd700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 0x7f57f0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f57f80096a0 tx=0x7f57f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:20.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.352+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57fc01ca30 con 0x7f58080601e0 2023-11-15T06:00:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:19.017578+0000 mgr.smithi003.cldzke (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:20.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.560+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58081080b0 con 0x7f58080601e0 2023-11-15T06:00:20.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.561+0000 7f57effff700 1 -- 172.21.15.137:0/1253677758 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f57fc01b9c0 con 0x7f58080601e0 2023-11-15T06:00:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:20.562 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:20.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 msgr2=0x7f57f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:20.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 0x7f57f0032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f57f80096a0 tx=0x7f57f8006b40).stop 2023-11-15T06:00:20.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 msgr2=0x7f58081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:20.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f57fc004dd0 tx=0x7f57fc009b30).stop 2023-11-15T06:00:20.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 shutdown_connections 2023-11-15T06:00:20.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f57f002ff60 0x7f57f0032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:20.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 --2- 172.21.15.137:0/1253677758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58080601e0 0x7f58081066e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:20.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 >> 172.21.15.137:0/1253677758 conn(0x7f58080f0b70 msgr2=0x7f58080f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:20.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 shutdown_connections 2023-11-15T06:00:20.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:20.563+0000 7f580e1af700 1 -- 172.21.15.137:0/1253677758 wait complete. 2023-11-15T06:00:20.565 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:21.932 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:20.561637+0000 mon.smithi003 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.137:0/1253677758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:21.933 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:21.017906+0000 mgr.smithi003.cldzke (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:21.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:21.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.310+0000 7ff60c20d700 1 -- 172.21.15.137:0/1562798711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 msgr2=0x7ff6040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.310+0000 7ff60c20d700 1 --2- 172.21.15.137:0/1562798711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff6040f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff5f4009fb0 tx=0x7ff5f4009b30).stop 2023-11-15T06:00:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.310+0000 7ff60c20d700 1 -- 172.21.15.137:0/1562798711 shutdown_connections 2023-11-15T06:00:23.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.310+0000 7ff60c20d700 1 --2- 172.21.15.137:0/1562798711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff6040f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:23.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.310+0000 7ff60c20d700 1 -- 172.21.15.137:0/1562798711 >> 172.21.15.137:0/1562798711 conn(0x7ff6040f1c70 msgr2=0x7ff6040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:23.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 -- 172.21.15.137:0/1562798711 shutdown_connections 2023-11-15T06:00:23.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 -- 172.21.15.137:0/1562798711 wait complete. 2023-11-15T06:00:23.315 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 Processor -- start 2023-11-15T06:00:23.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 -- start start 2023-11-15T06:00:23.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:23.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.311+0000 7ff60c20d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff604069e20 con 0x7ff6040f6510 2023-11-15T06:00:23.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff609fa9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:23.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff609fa9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37014/0 (socket says 172.21.15.137:37014) 2023-11-15T06:00:23.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff609fa9700 1 -- 172.21.15.137:0/4078630814 learned_addr learned my addr 172.21.15.137:0/4078630814 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:23.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff609fa9700 1 -- 172.21.15.137:0/4078630814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f4005040 con 0x7ff6040f6510 2023-11-15T06:00:23.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff609fa9700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff5f4013510 tx=0x7ff5f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:23.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5f4009320 con 0x7ff6040f6510 2023-11-15T06:00:23.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff604069fc0 con 0x7ff6040f6510 2023-11-15T06:00:23.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5f4013f00 con 0x7ff6040f6510 2023-11-15T06:00:23.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5f4014d90 con 0x7ff6040f6510 2023-11-15T06:00:23.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.312+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60406a450 con 0x7ff6040f6510 2023-11-15T06:00:23.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.313+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff604059100 con 0x7ff6040f6510 2023-11-15T06:00:23.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.313+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5f40095f0 con 0x7ff6040f6510 2023-11-15T06:00:23.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.313+0000 7ff5fa7fc700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 0x7ff5f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:23.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.313+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5f40410c0 con 0x7ff6040f6510 2023-11-15T06:00:23.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.313+0000 7ff6097a8700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 0x7ff5f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:23.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.314+0000 7ff6097a8700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 0x7ff5f0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff6000096a0 tx=0x7ff600006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:23.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.318+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5f4014420 con 0x7ff6040f6510 2023-11-15T06:00:23.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.524+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6040fbf60 con 0x7ff6040f6510 2023-11-15T06:00:23.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.524+0000 7ff5fa7fc700 1 -- 172.21.15.137:0/4078630814 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff5f401d7f0 con 0x7ff6040f6510 2023-11-15T06:00:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:23.527 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:23.527 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 msgr2=0x7ff5f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:23.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 0x7ff5f0032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff6000096a0 tx=0x7ff600006b40).stop 2023-11-15T06:00:23.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 msgr2=0x7ff60406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:23.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff5f4013510 tx=0x7ff5f400b040).stop 2023-11-15T06:00:23.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 shutdown_connections 2023-11-15T06:00:23.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff5f002ff60 0x7ff5f0032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:23.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 --2- 172.21.15.137:0/4078630814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff6040f6510 0x7ff60406b9a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:23.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 >> 172.21.15.137:0/4078630814 conn(0x7ff6040f1c70 msgr2=0x7ff6040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:23.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 shutdown_connections 2023-11-15T06:00:23.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:23.526+0000 7ff60c20d700 1 -- 172.21.15.137:0/4078630814 wait complete. 2023-11-15T06:00:23.530 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:23.018199+0000 mgr.smithi003.cldzke (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:23.525989+0000 mon.smithi003 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.137:0/4078630814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:24.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:24.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:26.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 -- 172.21.15.137:0/2281858062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 msgr2=0x7f33640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:26.033 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 --2- 172.21.15.137:0/2281858062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f33640f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f3354009fb0 tx=0x7f3354009b30).stop 2023-11-15T06:00:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 -- 172.21.15.137:0/2281858062 shutdown_connections 2023-11-15T06:00:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 --2- 172.21.15.137:0/2281858062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f33640f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 -- 172.21.15.137:0/2281858062 >> 172.21.15.137:0/2281858062 conn(0x7f33640f0b70 msgr2=0x7f33640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 -- 172.21.15.137:0/2281858062 shutdown_connections 2023-11-15T06:00:26.034 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.029+0000 7f336a9a9700 1 -- 172.21.15.137:0/2281858062 wait complete. 2023-11-15T06:00:26.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.030+0000 7f336a9a9700 1 Processor -- start 2023-11-15T06:00:26.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.030+0000 7f336a9a9700 1 -- start start 2023-11-15T06:00:26.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.030+0000 7f336a9a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:26.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.030+0000 7f336a9a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3364060240 con 0x7f33640f5410 2023-11-15T06:00:26.035 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3363fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3363fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37026/0 (socket says 172.21.15.137:37026) 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3363fff700 1 -- 172.21.15.137:0/3499611211 learned_addr learned my addr 172.21.15.137:0/3499611211 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3363fff700 1 -- 172.21.15.137:0/3499611211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3354005040 con 0x7f33640f5410 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3363fff700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f3354013510 tx=0x7f335400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3354009320 con 0x7f33640f5410 2023-11-15T06:00:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3364060440 con 0x7f33640f5410 2023-11-15T06:00:26.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3354013f00 con 0x7f33640f5410 2023-11-15T06:00:26.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3354014d90 con 0x7f33640f5410 2023-11-15T06:00:26.037 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.031+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3364109ae0 con 0x7f33640f5410 2023-11-15T06:00:26.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.032+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33540095f0 con 0x7f33640f5410 2023-11-15T06:00:26.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.032+0000 7f3360ff9700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 0x7f3350032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:26.038 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.032+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3354040fe0 con 0x7f33640f5410 2023-11-15T06:00:26.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.032+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3364045ad0 con 0x7f33640f5410 2023-11-15T06:00:26.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.033+0000 7f33637fe700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 0x7f3350032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:26.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.033+0000 7f33637fe700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 0x7f3350032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f334c0096a0 tx=0x7f334c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:26.039 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.037+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f335401d7f0 con 0x7f33640f5410 2023-11-15T06:00:26.251 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.250+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f336410a1f0 con 0x7f33640f5410 2023-11-15T06:00:26.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.250+0000 7f3360ff9700 1 -- 172.21.15.137:0/3499611211 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f335402b020 con 0x7f33640f5410 2023-11-15T06:00:26.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:26.253 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:26.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 msgr2=0x7f3350032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:26.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 0x7f3350032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f334c0096a0 tx=0x7f334c006b40).stop 2023-11-15T06:00:26.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 msgr2=0x7f3364109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:26.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f3354013510 tx=0x7f335400b040).stop 2023-11-15T06:00:26.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 shutdown_connections 2023-11-15T06:00:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f335002ff60 0x7f3350032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 --2- 172.21.15.137:0/3499611211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33640f5410 0x7f3364109430 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 >> 172.21.15.137:0/3499611211 conn(0x7f33640f0b70 msgr2=0x7f33640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 shutdown_connections 2023-11-15T06:00:26.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:26.252+0000 7f336a9a9700 1 -- 172.21.15.137:0/3499611211 wait complete. 2023-11-15T06:00:26.256 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:26.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:25.018518+0000 mgr.smithi003.cldzke (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:27.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:27 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:26.251738+0000 mon.smithi003 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.137:0/3499611211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:27.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:27.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:27.018819+0000 mgr.smithi003.cldzke (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:28.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 -- 172.21.15.137:0/3957278789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 msgr2=0x7f06080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:28.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 --2- 172.21.15.137:0/3957278789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f06080f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f05fc009fb0 tx=0x7f05fc009b30).stop 2023-11-15T06:00:28.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 -- 172.21.15.137:0/3957278789 shutdown_connections 2023-11-15T06:00:28.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 --2- 172.21.15.137:0/3957278789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f06080f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:28.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 -- 172.21.15.137:0/3957278789 >> 172.21.15.137:0/3957278789 conn(0x7f06080f0b70 msgr2=0x7f06080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:28.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 -- 172.21.15.137:0/3957278789 shutdown_connections 2023-11-15T06:00:28.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.750+0000 7f060eca3700 1 -- 172.21.15.137:0/3957278789 wait complete. 2023-11-15T06:00:28.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.751+0000 7f060eca3700 1 Processor -- start 2023-11-15T06:00:28.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.751+0000 7f060eca3700 1 -- start start 2023-11-15T06:00:28.755 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.751+0000 7f060eca3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:28.755 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.751+0000 7f060eca3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0608107240 con 0x7f06080f5410 2023-11-15T06:00:28.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060ca3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:28.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060ca3f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:37048/0 (socket says 172.21.15.137:37048) 2023-11-15T06:00:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060ca3f700 1 -- 172.21.15.137:0/618970845 learned_addr learned my addr 172.21.15.137:0/618970845 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060ca3f700 1 -- 172.21.15.137:0/618970845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc005040 con 0x7f06080f5410 2023-11-15T06:00:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060ca3f700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f05fc013510 tx=0x7f05fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05fc009320 con 0x7f06080f5410 2023-11-15T06:00:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05fc013f00 con 0x7f06080f5410 2023-11-15T06:00:28.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0608060220 con 0x7f06080f5410 2023-11-15T06:00:28.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05fc014d90 con 0x7f06080f5410 2023-11-15T06:00:28.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.752+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060810bd40 con 0x7f06080f5410 2023-11-15T06:00:28.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.753+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05fc0095f0 con 0x7f06080f5410 2023-11-15T06:00:28.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.753+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0608045ad0 con 0x7f06080f5410 2023-11-15T06:00:28.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.753+0000 7f06057fa700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 0x7f05f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:28.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.753+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05fc0410a0 con 0x7f06080f5410 2023-11-15T06:00:28.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.754+0000 7f0607fff700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 0x7f05f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:28.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.754+0000 7f0607fff700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 0x7f05f8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f05f40096a0 tx=0x7f05f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:28.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.758+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05fc01d7f0 con 0x7f06080f5410 2023-11-15T06:00:28.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.962+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06080fae60 con 0x7f06080f5410 2023-11-15T06:00:28.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.962+0000 7f06057fa700 1 -- 172.21.15.137:0/618970845 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f05fc010230 con 0x7f06080f5410 2023-11-15T06:00:28.964 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:28.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:28.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 msgr2=0x7f05f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:28.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 0x7f05f8032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f05f40096a0 tx=0x7f05f4006b40).stop 2023-11-15T06:00:28.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 msgr2=0x7f060810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:28.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f05fc013510 tx=0x7f05fc00b040).stop 2023-11-15T06:00:28.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 shutdown_connections 2023-11-15T06:00:28.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f05f802ff60 0x7f05f8032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:28.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 --2- 172.21.15.137:0/618970845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06080f5410 0x7f060810b850 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:28.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 >> 172.21.15.137:0/618970845 conn(0x7f06080f0b70 msgr2=0x7f06080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:28.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 shutdown_connections 2023-11-15T06:00:28.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:28.964+0000 7f060eca3700 1 -- 172.21.15.137:0/618970845 wait complete. 2023-11-15T06:00:28.968 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:29.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:29 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:28.963902+0000 mon.smithi003 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.137:0/618970845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:30.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:30.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:29.019170+0000 mgr.smithi003.cldzke (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:31.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 -- 172.21.15.137:0/2562915942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 msgr2=0x7f2ad00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:31.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/2562915942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad00f55c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f2ac4009fb0 tx=0x7f2ac4009b30).stop 2023-11-15T06:00:31.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 -- 172.21.15.137:0/2562915942 shutdown_connections 2023-11-15T06:00:31.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/2562915942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad00f55c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:31.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 -- 172.21.15.137:0/2562915942 >> 172.21.15.137:0/2562915942 conn(0x7f2ad00f09a0 msgr2=0x7f2ad00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:31.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 -- 172.21.15.137:0/2562915942 shutdown_connections 2023-11-15T06:00:31.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 -- 172.21.15.137:0/2562915942 wait complete. 2023-11-15T06:00:31.519 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.511+0000 7f2ad59ae700 1 Processor -- start 2023-11-15T06:00:31.520 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad59ae700 1 -- start start 2023-11-15T06:00:31.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad59ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:31.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad59ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad0108bb0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad49ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:31.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad49ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46508/0 (socket says 172.21.15.137:46508) 2023-11-15T06:00:31.521 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad49ac700 1 -- 172.21.15.137:0/1770514030 learned_addr learned my addr 172.21.15.137:0/1770514030 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:31.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad49ac700 1 -- 172.21.15.137:0/1770514030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4005040 con 0x7f2ad00f51a0 2023-11-15T06:00:31.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.512+0000 7f2ad49ac700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2ac4009830 tx=0x7f2ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:31.522 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.513+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac4009320 con 0x7f2ad00f51a0 2023-11-15T06:00:31.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.513+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad0108db0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.513+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ac40136c0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.523 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.513+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac401ee90 con 0x7f2ad00f51a0 2023-11-15T06:00:31.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.513+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad0109240 con 0x7f2ad00f51a0 2023-11-15T06:00:31.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ac401e420 con 0x7f2ad00f51a0 2023-11-15T06:00:31.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad0045ad0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.524 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2acd7fa700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 0x7f2ac00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:31.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ac4041010 con 0x7f2ad00f51a0 2023-11-15T06:00:31.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2acffff700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 0x7f2ac00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:31.525 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.514+0000 7f2acffff700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 0x7f2ac00323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2abc0096a0 tx=0x7f2abc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:31.526 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.518+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ac40095f0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.727+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ad010a0f0 con 0x7f2ad00f51a0 2023-11-15T06:00:31.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.727+0000 7f2acd7fa700 1 -- 172.21.15.137:0/1770514030 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2ac401cb10 con 0x7f2ad00f51a0 2023-11-15T06:00:31.729 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:31.730 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:31.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 msgr2=0x7f2ac00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:31.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 0x7f2ac00323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f2abc0096a0 tx=0x7f2abc006b40).stop 2023-11-15T06:00:31.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 msgr2=0x7f2ad01086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:31.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f2ac4009830 tx=0x7f2ac400b040).stop 2023-11-15T06:00:31.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 shutdown_connections 2023-11-15T06:00:31.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f2ac002ff10 0x7f2ac00323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:31.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 --2- 172.21.15.137:0/1770514030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ad00f51a0 0x7f2ad01086c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:31.732 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 >> 172.21.15.137:0/1770514030 conn(0x7f2ad00f09a0 msgr2=0x7f2ad005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:31.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 shutdown_connections 2023-11-15T06:00:31.733 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:31.729+0000 7f2ad59ae700 1 -- 172.21.15.137:0/1770514030 wait complete. 2023-11-15T06:00:31.733 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:31.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:31.019446+0000 mgr.smithi003.cldzke (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:32.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:31.728928+0000 mon.smithi003 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.137:0/1770514030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:33.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:33.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:33.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:33 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:33.019721+0000 mgr.smithi003.cldzke (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.295+0000 7fe50eb29700 1 -- 172.21.15.137:0/4232446993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 msgr2=0x7fe5080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.295+0000 7fe50eb29700 1 --2- 172.21.15.137:0/4232446993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5080f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe4fc009fb0 tx=0x7fe4fc009b30).stop 2023-11-15T06:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.295+0000 7fe50eb29700 1 -- 172.21.15.137:0/4232446993 shutdown_connections 2023-11-15T06:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.295+0000 7fe50eb29700 1 --2- 172.21.15.137:0/4232446993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5080f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.295+0000 7fe50eb29700 1 -- 172.21.15.137:0/4232446993 >> 172.21.15.137:0/4232446993 conn(0x7fe5080f0b70 msgr2=0x7fe5080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 -- 172.21.15.137:0/4232446993 shutdown_connections 2023-11-15T06:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 -- 172.21.15.137:0/4232446993 wait complete. 2023-11-15T06:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 Processor -- start 2023-11-15T06:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 -- start start 2023-11-15T06:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.296+0000 7fe50eb29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508060240 con 0x7fe5080f5410 2023-11-15T06:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50c8c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50c8c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46540/0 (socket says 172.21.15.137:46540) 2023-11-15T06:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50c8c5700 1 -- 172.21.15.137:0/1566545065 learned_addr learned my addr 172.21.15.137:0/1566545065 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50c8c5700 1 -- 172.21.15.137:0/1566545065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc005040 con 0x7fe5080f5410 2023-11-15T06:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50c8c5700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc009320 con 0x7fe5080f5410 2023-11-15T06:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe508060440 con 0x7fe5080f5410 2023-11-15T06:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc013f00 con 0x7fe5080f5410 2023-11-15T06:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc014d90 con 0x7fe5080f5410 2023-11-15T06:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.297+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe508109a80 con 0x7fe5080f5410 2023-11-15T06:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.298+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4fc0095f0 con 0x7fe5080f5410 2023-11-15T06:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.298+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe508045ad0 con 0x7fe5080f5410 2023-11-15T06:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.298+0000 7fe5057fa700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.298+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4fc0410c0 con 0x7fe5080f5410 2023-11-15T06:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.299+0000 7fe507fff700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.299+0000 7fe507fff700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 0x7fe4f8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe4f40096a0 tx=0x7fe4f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.303+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4fc014420 con 0x7fe5080f5410 2023-11-15T06:00:34.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.513+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe50810a2a0 con 0x7fe5080f5410 2023-11-15T06:00:34.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.513+0000 7fe5057fa700 1 -- 172.21.15.137:0/1566545065 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe4fc010230 con 0x7fe5080f5410 2023-11-15T06:00:34.515 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:34.515 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:34.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 msgr2=0x7fe4f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 0x7fe4f8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe4f40096a0 tx=0x7fe4f4006b40).stop 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 msgr2=0x7fe5081093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).stop 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 shutdown_connections 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:34.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 --2- 172.21.15.137:0/1566545065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5080f5410 0x7fe5081093d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:34.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 >> 172.21.15.137:0/1566545065 conn(0x7fe5080f0b70 msgr2=0x7fe5080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:34.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 shutdown_connections 2023-11-15T06:00:34.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:34.515+0000 7fe50eb29700 1 -- 172.21.15.137:0/1566545065 wait complete. 2023-11-15T06:00:34.519 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:34.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:34.514658+0000 mon.smithi003 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.137:0/1566545065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:35.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:35.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:35.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:35.020040+0000 mgr.smithi003.cldzke (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:35.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:35.115552+0000 mon.smithi003 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:00:35.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:35.118499+0000 mon.smithi003 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T06:00:37.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 -- 172.21.15.137:0/4127326427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 msgr2=0x7f9c480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:37.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/4127326427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c480f90f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f9c3c009fb0 tx=0x7f9c3c009b30).stop 2023-11-15T06:00:37.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 -- 172.21.15.137:0/4127326427 shutdown_connections 2023-11-15T06:00:37.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/4127326427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c480f90f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:37.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 -- 172.21.15.137:0/4127326427 >> 172.21.15.137:0/4127326427 conn(0x7f9c480f0b70 msgr2=0x7f9c480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 -- 172.21.15.137:0/4127326427 shutdown_connections 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.112+0000 7f9c4cc41700 1 -- 172.21.15.137:0/4127326427 wait complete. 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4cc41700 1 Processor -- start 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4cc41700 1 -- start start 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4cc41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:37.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4cc41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48106bd0 con 0x7f9c48060280 2023-11-15T06:00:37.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:37.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:46562/0 (socket says 172.21.15.137:46562) 2023-11-15T06:00:37.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.113+0000 7f9c4659c700 1 -- 172.21.15.137:0/3029228249 learned_addr learned my addr 172.21.15.137:0/3029228249 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:37.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c4659c700 1 -- 172.21.15.137:0/3029228249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c005040 con 0x7f9c48060280 2023-11-15T06:00:37.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c4659c700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f9c3c013510 tx=0x7f9c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:37.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c3c009320 con 0x7f9c48060280 2023-11-15T06:00:37.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c48106dd0 con 0x7f9c48060280 2023-11-15T06:00:37.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c3c013f00 con 0x7f9c48060280 2023-11-15T06:00:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c3c014dc0 con 0x7f9c48060280 2023-11-15T06:00:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.114+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c48107260 con 0x7f9c48060280 2023-11-15T06:00:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.115+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c3c01d3d0 con 0x7f9c48060280 2023-11-15T06:00:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.115+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48045ad0 con 0x7f9c48060280 2023-11-15T06:00:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.115+0000 7f9c36ffd700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 0x7f9c30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:37.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.115+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c3c041040 con 0x7f9c48060280 2023-11-15T06:00:37.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.115+0000 7f9c45d9b700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 0x7f9c30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:37.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.116+0000 7f9c45d9b700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 0x7f9c30032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9c380096a0 tx=0x7f9c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:37.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.120+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c3c0126f0 con 0x7f9c48060280 2023-11-15T06:00:37.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.326+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c48108110 con 0x7f9c48060280 2023-11-15T06:00:37.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.326+0000 7f9c36ffd700 1 -- 172.21.15.137:0/3029228249 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9c3c012330 con 0x7f9c48060280 2023-11-15T06:00:37.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:37.329 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:37.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 msgr2=0x7f9c30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:37.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 0x7f9c30032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9c380096a0 tx=0x7f9c38006b40).stop 2023-11-15T06:00:37.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 msgr2=0x7f9c481066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:37.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f9c3c013510 tx=0x7f9c3c00b040).stop 2023-11-15T06:00:37.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 shutdown_connections 2023-11-15T06:00:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9c3002ff60 0x7f9c30032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 --2- 172.21.15.137:0/3029228249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c48060280 0x7f9c481066e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 >> 172.21.15.137:0/3029228249 conn(0x7f9c480f0b70 msgr2=0x7f9c480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 shutdown_connections 2023-11-15T06:00:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:37.328+0000 7f9c4cc41700 1 -- 172.21.15.137:0/3029228249 wait complete. 2023-11-15T06:00:37.332 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:37.020442+0000 mgr.smithi003.cldzke (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:37.327936+0000 mon.smithi003 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.137:0/3029228249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:38.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:38.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:39.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- 172.21.15.137:0/1997514732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 msgr2=0x7fba2c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:39.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 --2- 172.21.15.137:0/1997514732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c0f55c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fba1c009fb0 tx=0x7fba1c009b30).stop 2023-11-15T06:00:39.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- 172.21.15.137:0/1997514732 shutdown_connections 2023-11-15T06:00:39.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 --2- 172.21.15.137:0/1997514732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c0f55c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:39.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- 172.21.15.137:0/1997514732 >> 172.21.15.137:0/1997514732 conn(0x7fba2c0f09a0 msgr2=0x7fba2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:39.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- 172.21.15.137:0/1997514732 shutdown_connections 2023-11-15T06:00:39.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- 172.21.15.137:0/1997514732 wait complete. 2023-11-15T06:00:39.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 Processor -- start 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- start start 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.791+0000 7fba334a1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c060240 con 0x7fba2c0f51a0 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba3249f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba3249f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:38068/0 (socket says 172.21.15.137:38068) 2023-11-15T06:00:39.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba3249f700 1 -- 172.21.15.137:0/2423550233 learned_addr learned my addr 172.21.15.137:0/2423550233 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:39.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba3249f700 1 -- 172.21.15.137:0/2423550233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c005040 con 0x7fba2c0f51a0 2023-11-15T06:00:39.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba3249f700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:39.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c009320 con 0x7fba2c0f51a0 2023-11-15T06:00:39.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba1c013f00 con 0x7fba2c0f51a0 2023-11-15T06:00:39.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c060440 con 0x7fba2c0f51a0 2023-11-15T06:00:39.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba1c014dc0 con 0x7fba2c0f51a0 2023-11-15T06:00:39.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.792+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c109860 con 0x7fba2c0f51a0 2023-11-15T06:00:39.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.793+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba1c0095f0 con 0x7fba2c0f51a0 2023-11-15T06:00:39.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.793+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c045ad0 con 0x7fba2c0f51a0 2023-11-15T06:00:39.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.793+0000 7fba22ffd700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 0x7fba18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:39.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.793+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba1c041040 con 0x7fba2c0f51a0 2023-11-15T06:00:39.800 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.793+0000 7fba31c9e700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 0x7fba18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:39.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.794+0000 7fba31c9e700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 0x7fba18032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fba280096a0 tx=0x7fba28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:39.801 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:39.798+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba1c014420 con 0x7fba2c0f51a0 2023-11-15T06:00:40.009 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.008+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba2c10a080 con 0x7fba2c0f51a0 2023-11-15T06:00:40.011 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.008+0000 7fba22ffd700 1 -- 172.21.15.137:0/2423550233 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fba1c010230 con 0x7fba2c0f51a0 2023-11-15T06:00:40.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:40.012 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:40.013 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 msgr2=0x7fba18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 0x7fba18032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fba280096a0 tx=0x7fba28006b40).stop 2023-11-15T06:00:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 msgr2=0x7fba2c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fba1c013510 tx=0x7fba1c00b040).stop 2023-11-15T06:00:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 shutdown_connections 2023-11-15T06:00:40.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fba1802ff60 0x7fba18032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:40.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 --2- 172.21.15.137:0/2423550233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba2c0f51a0 0x7fba2c1091b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:40.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 >> 172.21.15.137:0/2423550233 conn(0x7fba2c0f09a0 msgr2=0x7fba2c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:40.015 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 shutdown_connections 2023-11-15T06:00:40.016 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:40.010+0000 7fba334a1700 1 -- 172.21.15.137:0/2423550233 wait complete. 2023-11-15T06:00:40.016 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:40.349 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:39.020811+0000 mgr.smithi003.cldzke (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:40.350 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:40.009797+0000 mon.smithi003 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.137:0/2423550233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:41.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:41.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:41.021155+0000 mgr.smithi003.cldzke (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:42.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 -- 172.21.15.137:0/3354725232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 msgr2=0x7fada80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:42.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 --2- 172.21.15.137:0/3354725232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada80f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fad90009fb0 tx=0x7fad90009b30).stop 2023-11-15T06:00:42.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 -- 172.21.15.137:0/3354725232 shutdown_connections 2023-11-15T06:00:42.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 --2- 172.21.15.137:0/3354725232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada80f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:42.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 -- 172.21.15.137:0/3354725232 >> 172.21.15.137:0/3354725232 conn(0x7fada80f0b70 msgr2=0x7fada80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:42.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 -- 172.21.15.137:0/3354725232 shutdown_connections 2023-11-15T06:00:42.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.557+0000 7fadacba7700 1 -- 172.21.15.137:0/3354725232 wait complete. 2023-11-15T06:00:42.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fadacba7700 1 Processor -- start 2023-11-15T06:00:42.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fadacba7700 1 -- start start 2023-11-15T06:00:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fadacba7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fadacba7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8060220 con 0x7fada80f5410 2023-11-15T06:00:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fada659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fada659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:38080/0 (socket says 172.21.15.137:38080) 2023-11-15T06:00:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.558+0000 7fada659c700 1 -- 172.21.15.137:0/1140822247 learned_addr learned my addr 172.21.15.137:0/1140822247 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:42.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fada659c700 1 -- 172.21.15.137:0/1140822247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad90005040 con 0x7fada80f5410 2023-11-15T06:00:42.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fada659c700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fad90013510 tx=0x7fad9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:42.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad90009320 con 0x7fada80f5410 2023-11-15T06:00:42.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad90013f00 con 0x7fada80f5410 2023-11-15T06:00:42.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada8060420 con 0x7fada80f5410 2023-11-15T06:00:42.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad90014d90 con 0x7fada80f5410 2023-11-15T06:00:42.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.559+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada8109b00 con 0x7fada80f5410 2023-11-15T06:00:42.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.560+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad900095f0 con 0x7fada80f5410 2023-11-15T06:00:42.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.560+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8059100 con 0x7fada80f5410 2023-11-15T06:00:42.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.560+0000 7fad9effd700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 0x7fad94032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:42.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.560+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad90040f90 con 0x7fada80f5410 2023-11-15T06:00:42.567 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.560+0000 7fada5d9b700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 0x7fad94032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:42.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.561+0000 7fada5d9b700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 0x7fad94032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fad980096a0 tx=0x7fad98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:42.568 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.565+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad90014420 con 0x7fada80f5410 2023-11-15T06:00:42.769 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.768+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fada8045ad0 con 0x7fada80f5410 2023-11-15T06:00:42.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.769+0000 7fad9effd700 1 -- 172.21.15.137:0/1140822247 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fad90010230 con 0x7fada80f5410 2023-11-15T06:00:42.770 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:42.771 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:42.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 msgr2=0x7fad94032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:42.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 0x7fad94032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fad980096a0 tx=0x7fad98006b40).stop 2023-11-15T06:00:42.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 msgr2=0x7fada8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:42.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fad90013510 tx=0x7fad9000b040).stop 2023-11-15T06:00:42.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 shutdown_connections 2023-11-15T06:00:42.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fad9402ffb0 0x7fad94032470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:42.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 --2- 172.21.15.137:0/1140822247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fada80f5410 0x7fada8109450 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:42.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 >> 172.21.15.137:0/1140822247 conn(0x7fada80f0b70 msgr2=0x7fada80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:42.773 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 shutdown_connections 2023-11-15T06:00:42.774 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:42.770+0000 7fadacba7700 1 -- 172.21.15.137:0/1140822247 wait complete. 2023-11-15T06:00:42.774 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:43.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:42.769977+0000 mon.smithi003 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.137:0/1140822247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:44.158 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:44.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:44.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:43 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:43.021468+0000 mgr.smithi003.cldzke (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:45.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 -- 172.21.15.137:0/2085664410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 msgr2=0x7f31200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:45.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 --2- 172.21.15.137:0/2085664410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f31200f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f3114009fb0 tx=0x7f3114009b30).stop 2023-11-15T06:00:45.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 -- 172.21.15.137:0/2085664410 shutdown_connections 2023-11-15T06:00:45.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 --2- 172.21.15.137:0/2085664410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f31200f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:45.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 -- 172.21.15.137:0/2085664410 >> 172.21.15.137:0/2085664410 conn(0x7f31200f1c70 msgr2=0x7f31200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:45.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 -- 172.21.15.137:0/2085664410 shutdown_connections 2023-11-15T06:00:45.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.323+0000 7f3125f84700 1 -- 172.21.15.137:0/2085664410 wait complete. 2023-11-15T06:00:45.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f3125f84700 1 Processor -- start 2023-11-15T06:00:45.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f3125f84700 1 -- start start 2023-11-15T06:00:45.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f3125f84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:45.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f3125f84700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3120069e20 con 0x7f31200f6510 2023-11-15T06:00:45.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f311f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:45.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f311f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:38100/0 (socket says 172.21.15.137:38100) 2023-11-15T06:00:45.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.324+0000 7f311f7fe700 1 -- 172.21.15.137:0/2480797129 learned_addr learned my addr 172.21.15.137:0/2480797129 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:45.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f311f7fe700 1 -- 172.21.15.137:0/2480797129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114005040 con 0x7f31200f6510 2023-11-15T06:00:45.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f311f7fe700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3114013510 tx=0x7f311400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:45.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3114009320 con 0x7f31200f6510 2023-11-15T06:00:45.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3120069fc0 con 0x7f31200f6510 2023-11-15T06:00:45.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3114013f00 con 0x7f31200f6510 2023-11-15T06:00:45.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3114014d90 con 0x7f31200f6510 2023-11-15T06:00:45.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.325+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312006a450 con 0x7f31200f6510 2023-11-15T06:00:45.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.326+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3120059100 con 0x7f31200f6510 2023-11-15T06:00:45.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.326+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31140095f0 con 0x7f31200f6510 2023-11-15T06:00:45.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.326+0000 7f3107fff700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 0x7f3108032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:45.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.326+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3114041000 con 0x7f31200f6510 2023-11-15T06:00:45.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.326+0000 7f311effd700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 0x7f3108032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:45.332 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.327+0000 7f311effd700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 0x7f3108032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f31100096a0 tx=0x7f3110006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:45.333 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.331+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3114014420 con 0x7f31200f6510 2023-11-15T06:00:45.543 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.542+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31200fbf60 con 0x7f31200f6510 2023-11-15T06:00:45.544 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.542+0000 7f3107fff700 1 -- 172.21.15.137:0/2480797129 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3114010230 con 0x7f31200f6510 2023-11-15T06:00:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:45.545 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:45.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 msgr2=0x7f3108032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:45.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 0x7f3108032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f31100096a0 tx=0x7f3110006b40).stop 2023-11-15T06:00:45.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 msgr2=0x7f312006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:45.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f3114013510 tx=0x7f311400b040).stop 2023-11-15T06:00:45.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 shutdown_connections 2023-11-15T06:00:45.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f310802ff60 0x7f3108032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:45.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 --2- 172.21.15.137:0/2480797129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f31200f6510 0x7f312006b9a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:45.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 >> 172.21.15.137:0/2480797129 conn(0x7f31200f1c70 msgr2=0x7f31200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:45.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 shutdown_connections 2023-11-15T06:00:45.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:45.544+0000 7f3125f84700 1 -- 172.21.15.137:0/2480797129 wait complete. 2023-11-15T06:00:45.548 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:45.021847+0000 mgr.smithi003.cldzke (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:45.543819+0000 mon.smithi003 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.137:0/2480797129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:46.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:46.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:48.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.122+0000 7f14c7fff700 1 -- 172.21.15.137:0/2688007592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 msgr2=0x7f14c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:48.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 --2- 172.21.15.137:0/2688007592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c80f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f14b8009fb0 tx=0x7f14b8009b30).stop 2023-11-15T06:00:48.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- 172.21.15.137:0/2688007592 shutdown_connections 2023-11-15T06:00:48.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 --2- 172.21.15.137:0/2688007592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c80f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:48.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- 172.21.15.137:0/2688007592 >> 172.21.15.137:0/2688007592 conn(0x7f14c80f09a0 msgr2=0x7f14c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:48.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- 172.21.15.137:0/2688007592 shutdown_connections 2023-11-15T06:00:48.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- 172.21.15.137:0/2688007592 wait complete. 2023-11-15T06:00:48.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 Processor -- start 2023-11-15T06:00:48.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- start start 2023-11-15T06:00:48.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:48.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.123+0000 7f14c7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c8060220 con 0x7f14c80f51a0 2023-11-15T06:00:48.128 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:48.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:38124/0 (socket says 172.21.15.137:38124) 2023-11-15T06:00:48.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c6ffd700 1 -- 172.21.15.137:0/1580921364 learned_addr learned my addr 172.21.15.137:0/1580921364 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:48.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c6ffd700 1 -- 172.21.15.137:0/1580921364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b8005040 con 0x7f14c80f51a0 2023-11-15T06:00:48.129 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c6ffd700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f14b8013510 tx=0x7f14b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:48.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b8009320 con 0x7f14c80f51a0 2023-11-15T06:00:48.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c8060420 con 0x7f14c80f51a0 2023-11-15T06:00:48.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14b8013f00 con 0x7f14c80f51a0 2023-11-15T06:00:48.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b8014dc0 con 0x7f14c80f51a0 2023-11-15T06:00:48.130 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.124+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c8109880 con 0x7f14c80f51a0 2023-11-15T06:00:48.131 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.125+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14b80095f0 con 0x7f14c80f51a0 2023-11-15T06:00:48.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.125+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c8059100 con 0x7f14c80f51a0 2023-11-15T06:00:48.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.125+0000 7f14af7fe700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:48.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.125+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14b8041000 con 0x7f14c80f51a0 2023-11-15T06:00:48.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.126+0000 7f14c67fc700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:48.132 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.126+0000 7f14c67fc700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 0x7f14b0032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f14bc0096a0 tx=0x7f14bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:48.133 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.131+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14b8014420 con 0x7f14c80f51a0 2023-11-15T06:00:48.336 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.335+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14c8045ad0 con 0x7f14c80f51a0 2023-11-15T06:00:48.337 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.335+0000 7f14af7fe700 1 -- 172.21.15.137:0/1580921364 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f14b8010230 con 0x7f14c80f51a0 2023-11-15T06:00:48.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:48.338 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 msgr2=0x7f14b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 0x7f14b0032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f14bc0096a0 tx=0x7f14bc006b40).stop 2023-11-15T06:00:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 msgr2=0x7f14c81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f14b8013510 tx=0x7f14b800b040).stop 2023-11-15T06:00:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 shutdown_connections 2023-11-15T06:00:48.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:48.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 --2- 172.21.15.137:0/1580921364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14c80f51a0 0x7f14c81091d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:48.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 >> 172.21.15.137:0/1580921364 conn(0x7f14c80f09a0 msgr2=0x7f14c80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:48.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 shutdown_connections 2023-11-15T06:00:48.340 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:48.337+0000 7f14c7fff700 1 -- 172.21.15.137:0/1580921364 wait complete. 2023-11-15T06:00:48.341 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:48.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:47.022188+0000 mgr.smithi003.cldzke (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:49.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:49 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:48.336852+0000 mon.smithi003 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.137:0/1580921364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:49.733 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:49.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:50.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:49.022556+0000 mgr.smithi003.cldzke (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:50.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- 172.21.15.137:0/2527776143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 msgr2=0x7f8a140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:50.932 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 --2- 172.21.15.137:0/2527776143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a140f66c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f8a04009fb0 tx=0x7f8a04009b30).stop 2023-11-15T06:00:50.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- 172.21.15.137:0/2527776143 shutdown_connections 2023-11-15T06:00:50.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 --2- 172.21.15.137:0/2527776143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a140f66c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:50.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- 172.21.15.137:0/2527776143 >> 172.21.15.137:0/2527776143 conn(0x7f8a140f1ac0 msgr2=0x7f8a140f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:50.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- 172.21.15.137:0/2527776143 shutdown_connections 2023-11-15T06:00:50.933 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- 172.21.15.137:0/2527776143 wait complete. 2023-11-15T06:00:50.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 Processor -- start 2023-11-15T06:00:50.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.930+0000 7f8a18968700 1 -- start start 2023-11-15T06:00:50.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a18968700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:50.934 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a18968700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a14106990 con 0x7f8a140f62a0 2023-11-15T06:00:50.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:50.936 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a137fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58392/0 (socket says 172.21.15.137:58392) 2023-11-15T06:00:50.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a137fe700 1 -- 172.21.15.137:0/3129362011 learned_addr learned my addr 172.21.15.137:0/3129362011 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:50.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a137fe700 1 -- 172.21.15.137:0/3129362011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a04005040 con 0x7f8a140f62a0 2023-11-15T06:00:50.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a137fe700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f8a04013510 tx=0x7f8a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:50.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a04009320 con 0x7f8a140f62a0 2023-11-15T06:00:50.937 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a14106b90 con 0x7f8a140f62a0 2023-11-15T06:00:50.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.931+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a04013f00 con 0x7f8a140f62a0 2023-11-15T06:00:50.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.932+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a04014dc0 con 0x7f8a140f62a0 2023-11-15T06:00:50.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.932+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a14107020 con 0x7f8a140f62a0 2023-11-15T06:00:50.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.932+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a040095f0 con 0x7f8a140f62a0 2023-11-15T06:00:50.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.932+0000 7f89fbfff700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 0x7f89fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.933+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a04041000 con 0x7f8a140f62a0 2023-11-15T06:00:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.932+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a14045ad0 con 0x7f8a140f62a0 2023-11-15T06:00:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.933+0000 7f8a12ffd700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 0x7f89fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.933+0000 7f8a12ffd700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 0x7f89fc032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:50.940 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:50.937+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a0401c790 con 0x7f8a140f62a0 2023-11-15T06:00:51.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.141+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a140fbdd0 con 0x7f8a140f62a0 2023-11-15T06:00:51.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.142+0000 7f89fbfff700 1 -- 172.21.15.137:0/3129362011 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8a0402b020 con 0x7f8a140f62a0 2023-11-15T06:00:51.144 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:51.144 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:51.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 msgr2=0x7f89fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:51.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 0x7f89fc032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).stop 2023-11-15T06:00:51.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 msgr2=0x7f8a141064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:51.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f8a04013510 tx=0x7f8a0400b040).stop 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 shutdown_connections 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f89fc02ff60 0x7f89fc032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 --2- 172.21.15.137:0/3129362011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a140f62a0 0x7f8a141064a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 >> 172.21.15.137:0/3129362011 conn(0x7f8a140f1ac0 msgr2=0x7f8a140f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 shutdown_connections 2023-11-15T06:00:51.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:51.144+0000 7f8a18968700 1 -- 172.21.15.137:0/3129362011 wait complete. 2023-11-15T06:00:51.148 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:51.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:51.022946+0000 mgr.smithi003.cldzke (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:51.143726+0000 mon.smithi003 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.137:0/3129362011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:52.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:52.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:53.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 -- 172.21.15.137:0/1483057476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 msgr2=0x7f30940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:53.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 --2- 172.21.15.137:0/1483057476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f3084009fb0 tx=0x7f3084009b30).stop 2023-11-15T06:00:53.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 -- 172.21.15.137:0/1483057476 shutdown_connections 2023-11-15T06:00:53.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 --2- 172.21.15.137:0/1483057476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:53.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 -- 172.21.15.137:0/1483057476 >> 172.21.15.137:0/1483057476 conn(0x7f30940f0b70 msgr2=0x7f30940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:53.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 -- 172.21.15.137:0/1483057476 shutdown_connections 2023-11-15T06:00:53.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.704+0000 7f3099103700 1 -- 172.21.15.137:0/1483057476 wait complete. 2023-11-15T06:00:53.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.705+0000 7f3099103700 1 Processor -- start 2023-11-15T06:00:53.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.705+0000 7f3099103700 1 -- start start 2023-11-15T06:00:53.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.705+0000 7f3099103700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:53.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.705+0000 7f3099103700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940fb820 con 0x7f30940f5410 2023-11-15T06:00:53.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.705+0000 7f3092d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:53.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3092d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58412/0 (socket says 172.21.15.137:58412) 2023-11-15T06:00:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3092d9d700 1 -- 172.21.15.137:0/3549045140 learned_addr learned my addr 172.21.15.137:0/3549045140 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3092d9d700 1 -- 172.21.15.137:0/3549045140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084005040 con 0x7f30940f5410 2023-11-15T06:00:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3092d9d700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3084013510 tx=0x7f308400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3084009320 con 0x7f30940f5410 2023-11-15T06:00:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30940fb9c0 con 0x7f30940f5410 2023-11-15T06:00:53.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3084013f00 con 0x7f30940f5410 2023-11-15T06:00:53.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3084014d90 con 0x7f30940f5410 2023-11-15T06:00:53.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.706+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30940fbe50 con 0x7f30940f5410 2023-11-15T06:00:53.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.707+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3094059100 con 0x7f30940f5410 2023-11-15T06:00:53.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.707+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30840095f0 con 0x7f30940f5410 2023-11-15T06:00:53.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.707+0000 7f307b7fe700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 0x7f307c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:53.713 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.707+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3084041080 con 0x7f30940f5410 2023-11-15T06:00:53.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.707+0000 7f309259c700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 0x7f307c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:53.714 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.708+0000 7f309259c700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 0x7f307c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f30880096a0 tx=0x7f3088006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:53.715 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.712+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3084014420 con 0x7f30940f5410 2023-11-15T06:00:53.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.917+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3094045ad0 con 0x7f30940f5410 2023-11-15T06:00:53.920 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.918+0000 7f307b7fe700 1 -- 172.21.15.137:0/3549045140 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3084010230 con 0x7f30940f5410 2023-11-15T06:00:53.920 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:53.921 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:53.921 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.919+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 msgr2=0x7f307c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.919+0000 7f3099103700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 0x7f307c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f30880096a0 tx=0x7f3088006b40).stop 2023-11-15T06:00:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.919+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 msgr2=0x7f30940fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.919+0000 7f3099103700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f3084013510 tx=0x7f308400b040).stop 2023-11-15T06:00:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 shutdown_connections 2023-11-15T06:00:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f307c02ff60 0x7f307c032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 --2- 172.21.15.137:0/3549045140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30940f5410 0x7f30940fd3a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 >> 172.21.15.137:0/3549045140 conn(0x7f30940f0b70 msgr2=0x7f30940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 shutdown_connections 2023-11-15T06:00:53.923 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:53.920+0000 7f3099103700 1 -- 172.21.15.137:0/3549045140 wait complete. 2023-11-15T06:00:53.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:54.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:53.023330+0000 mgr.smithi003.cldzke (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:54.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:53.919409+0000 mon.smithi003 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.137:0/3549045140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:55.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:55.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:55.023713+0000 mgr.smithi003.cldzke (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 -- 172.21.15.137:0/3240576549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 msgr2=0x7f17480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 --2- 172.21.15.137:0/3240576549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f17480f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f173c009fb0 tx=0x7f173c009b30).stop 2023-11-15T06:00:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 -- 172.21.15.137:0/3240576549 shutdown_connections 2023-11-15T06:00:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 --2- 172.21.15.137:0/3240576549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f17480f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 -- 172.21.15.137:0/3240576549 >> 172.21.15.137:0/3240576549 conn(0x7f17480f0b70 msgr2=0x7f17480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 -- 172.21.15.137:0/3240576549 shutdown_connections 2023-11-15T06:00:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.508+0000 7f174cd2c700 1 -- 172.21.15.137:0/3240576549 wait complete. 2023-11-15T06:00:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174cd2c700 1 Processor -- start 2023-11-15T06:00:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174cd2c700 1 -- start start 2023-11-15T06:00:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174cd2c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:56.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174cd2c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1748105b20 con 0x7f17480f5410 2023-11-15T06:00:56.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:56.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58432/0 (socket says 172.21.15.137:58432) 2023-11-15T06:00:56.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174659c700 1 -- 172.21.15.137:0/1870848988 learned_addr learned my addr 172.21.15.137:0/1870848988 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:56.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.509+0000 7f174659c700 1 -- 172.21.15.137:0/1870848988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c005040 con 0x7f17480f5410 2023-11-15T06:00:56.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f174659c700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f173c013510 tx=0x7f173c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:56.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173c009320 con 0x7f17480f5410 2023-11-15T06:00:56.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1748105d20 con 0x7f17480f5410 2023-11-15T06:00:56.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f173c013f00 con 0x7f17480f5410 2023-11-15T06:00:56.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f173c014d90 con 0x7f17480f5410 2023-11-15T06:00:56.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.510+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17481061b0 con 0x7f17480f5410 2023-11-15T06:00:56.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1748045ad0 con 0x7f17480f5410 2023-11-15T06:00:56.516 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f173c014420 con 0x7f17480f5410 2023-11-15T06:00:56.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f1736ffd700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 0x7f1730032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:56.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f173c040fe0 con 0x7f17480f5410 2023-11-15T06:00:56.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f1745d9b700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 0x7f1730032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:56.517 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.511+0000 7f1745d9b700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 0x7f1730032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f17380096a0 tx=0x7f1738006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:56.518 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.515+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f173c06d2b0 con 0x7f17480f5410 2023-11-15T06:00:56.723 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.722+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17480fae60 con 0x7f17480f5410 2023-11-15T06:00:56.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.723+0000 7f1736ffd700 1 -- 172.21.15.137:0/1870848988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f173c0095f0 con 0x7f17480f5410 2023-11-15T06:00:56.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:56.725 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:56.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 msgr2=0x7f1730032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:56.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 0x7f1730032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f17380096a0 tx=0x7f1738006b40).stop 2023-11-15T06:00:56.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 msgr2=0x7f1748105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:56.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f173c013510 tx=0x7f173c00b040).stop 2023-11-15T06:00:56.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 shutdown_connections 2023-11-15T06:00:56.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f173002ffb0 0x7f1730032470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:56.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 --2- 172.21.15.137:0/1870848988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17480f5410 0x7f1748105630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:56.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 >> 172.21.15.137:0/1870848988 conn(0x7f17480f0b70 msgr2=0x7f17480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:56.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 shutdown_connections 2023-11-15T06:00:56.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:56.724+0000 7f174cd2c700 1 -- 172.21.15.137:0/1870848988 wait complete. 2023-11-15T06:00:56.728 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:00:57.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:57 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:56.723927+0000 mon.smithi003 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.137:0/1870848988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:00:58.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:00:58.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:00:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:00:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:57.024009+0000 mgr.smithi003.cldzke (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:00:59.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.349+0000 7f55d0a75700 1 -- 172.21.15.137:0/113489782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 msgr2=0x7f55cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:59.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.349+0000 7f55d0a75700 1 --2- 172.21.15.137:0/113489782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc0f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f55b4009fb0 tx=0x7f55b4009b30).stop 2023-11-15T06:00:59.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- 172.21.15.137:0/113489782 shutdown_connections 2023-11-15T06:00:59.351 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 --2- 172.21.15.137:0/113489782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc0f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:59.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- 172.21.15.137:0/113489782 >> 172.21.15.137:0/113489782 conn(0x7f55cc0f0b70 msgr2=0x7f55cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:59.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- 172.21.15.137:0/113489782 shutdown_connections 2023-11-15T06:00:59.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- 172.21.15.137:0/113489782 wait complete. 2023-11-15T06:00:59.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 Processor -- start 2023-11-15T06:00:59.352 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- start start 2023-11-15T06:00:59.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:59.353 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55d0a75700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc060240 con 0x7f55cc0f5410 2023-11-15T06:00:59.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.350+0000 7f55ca59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:59.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55ca59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:51794/0 (socket says 172.21.15.137:51794) 2023-11-15T06:00:59.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55ca59c700 1 -- 172.21.15.137:0/1680928744 learned_addr learned my addr 172.21.15.137:0/1680928744 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:00:59.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55ca59c700 1 -- 172.21.15.137:0/1680928744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4005040 con 0x7f55cc0f5410 2023-11-15T06:00:59.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55ca59c700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f55b4013510 tx=0x7f55b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:59.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b4009320 con 0x7f55cc0f5410 2023-11-15T06:00:59.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc060440 con 0x7f55cc0f5410 2023-11-15T06:00:59.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55b4013f00 con 0x7f55cc0f5410 2023-11-15T06:00:59.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55b4014d90 con 0x7f55cc0f5410 2023-11-15T06:00:59.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.351+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc109ae0 con 0x7f55cc0f5410 2023-11-15T06:00:59.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.352+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55b40095f0 con 0x7f55cc0f5410 2023-11-15T06:00:59.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.352+0000 7f55c2ffd700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 0x7f55b8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:00:59.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.352+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc045ad0 con 0x7f55cc0f5410 2023-11-15T06:00:59.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.352+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55b4040fe0 con 0x7f55cc0f5410 2023-11-15T06:00:59.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.352+0000 7f55c9d9b700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 0x7f55b8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:00:59.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.353+0000 7f55c9d9b700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 0x7f55b8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f55bc0096a0 tx=0x7f55bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:00:59.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.357+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55b401d7f0 con 0x7f55cc0f5410 2023-11-15T06:00:59.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.561+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55cc0faf50 con 0x7f55cc0f5410 2023-11-15T06:00:59.563 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.562+0000 7f55c2ffd700 1 -- 172.21.15.137:0/1680928744 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f55b4010230 con 0x7f55cc0f5410 2023-11-15T06:00:59.563 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:00:59.563 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:00:59.564 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 msgr2=0x7f55b8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 0x7f55b8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f55bc0096a0 tx=0x7f55bc006b40).stop 2023-11-15T06:00:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 msgr2=0x7f55cc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:00:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f55b4013510 tx=0x7f55b400b040).stop 2023-11-15T06:00:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 shutdown_connections 2023-11-15T06:00:59.565 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f55b802ffb0 0x7f55b8032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 --2- 172.21.15.137:0/1680928744 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f55cc0f5410 0x7f55cc109430 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:00:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 >> 172.21.15.137:0/1680928744 conn(0x7f55cc0f0b70 msgr2=0x7f55cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:00:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 shutdown_connections 2023-11-15T06:00:59.566 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:00:59.564+0000 7f55d0a75700 1 -- 172.21.15.137:0/1680928744 wait complete. 2023-11-15T06:00:59.567 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:00:59.024310+0000 mgr.smithi003.cldzke (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:00:59.562521+0000 mon.smithi003 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.137:0/1680928744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:00.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:00.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:01.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:01.024630+0000 mgr.smithi003.cldzke (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 -- 172.21.15.137:0/3041446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 msgr2=0x7f02fc0f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 --2- 172.21.15.137:0/3041446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc0f6db0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f02f0009fb0 tx=0x7f02f0009b30).stop 2023-11-15T06:01:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 -- 172.21.15.137:0/3041446616 shutdown_connections 2023-11-15T06:01:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 --2- 172.21.15.137:0/3041446616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc0f6db0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 -- 172.21.15.137:0/3041446616 >> 172.21.15.137:0/3041446616 conn(0x7f02fc0f0260 msgr2=0x7f02fc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 -- 172.21.15.137:0/3041446616 shutdown_connections 2023-11-15T06:01:02.091 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.088+0000 7f0301940700 1 -- 172.21.15.137:0/3041446616 wait complete. 2023-11-15T06:01:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f0301940700 1 Processor -- start 2023-11-15T06:01:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f0301940700 1 -- start start 2023-11-15T06:01:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f0301940700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:02.093 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f0301940700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc106c20 con 0x7f02fc0f6990 2023-11-15T06:01:02.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f02faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:02.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f02faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:51820/0 (socket says 172.21.15.137:51820) 2023-11-15T06:01:02.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.089+0000 7f02faffd700 1 -- 172.21.15.137:0/3528742971 learned_addr learned my addr 172.21.15.137:0/3528742971 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:02.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f02faffd700 1 -- 172.21.15.137:0/3528742971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0005040 con 0x7f02fc0f6990 2023-11-15T06:01:02.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f02faffd700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f02f0013510 tx=0x7f02f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f0009320 con 0x7f02fc0f6990 2023-11-15T06:01:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc106e20 con 0x7f02fc0f6990 2023-11-15T06:01:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02f0013f00 con 0x7f02fc0f6990 2023-11-15T06:01:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f0014d90 con 0x7f02fc0f6990 2023-11-15T06:01:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.090+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc1072b0 con 0x7f02fc0f6990 2023-11-15T06:01:02.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.091+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02f00095f0 con 0x7f02fc0f6990 2023-11-15T06:01:02.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.091+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc045ad0 con 0x7f02fc0f6990 2023-11-15T06:01:02.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.091+0000 7f02e37fe700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 0x7f02e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:02.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.091+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02f00410c0 con 0x7f02fc0f6990 2023-11-15T06:01:02.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.091+0000 7f02fa7fc700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 0x7f02e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:02.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.092+0000 7f02fa7fc700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 0x7f02e4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:02.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.096+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02f0014420 con 0x7f02fc0f6990 2023-11-15T06:01:02.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.302+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02fc0fae60 con 0x7f02fc0f6990 2023-11-15T06:01:02.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.303+0000 7f02e37fe700 1 -- 172.21.15.137:0/3528742971 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f02f0010230 con 0x7f02fc0f6990 2023-11-15T06:01:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:02.304 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:02.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.304+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 msgr2=0x7f02e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:02.305 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.304+0000 7f0301940700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 0x7f02e4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).stop 2023-11-15T06:01:02.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.304+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 msgr2=0x7f02fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:02.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.304+0000 7f0301940700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f02f0013510 tx=0x7f02f000b040).stop 2023-11-15T06:01:02.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 shutdown_connections 2023-11-15T06:01:02.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f02e402ff60 0x7f02e4032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:02.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 --2- 172.21.15.137:0/3528742971 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f6990 0x7f02fc106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:02.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 >> 172.21.15.137:0/3528742971 conn(0x7f02fc0f0260 msgr2=0x7f02fc0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:02.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 shutdown_connections 2023-11-15T06:01:02.307 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:02.305+0000 7f0301940700 1 -- 172.21.15.137:0/3528742971 wait complete. 2023-11-15T06:01:02.307 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:02.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:02.303323+0000 mon.smithi003 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.137:0/3528742971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:03.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:03.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:03.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:03.024959+0000 mgr.smithi003.cldzke (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:04.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 -- 172.21.15.137:0/1510191534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 msgr2=0x7f1e540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:04.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 --2- 172.21.15.137:0/1510191534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e540f55c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1e44009fb0 tx=0x7f1e44009b30).stop 2023-11-15T06:01:04.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 -- 172.21.15.137:0/1510191534 shutdown_connections 2023-11-15T06:01:04.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 --2- 172.21.15.137:0/1510191534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e540f55c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:04.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 -- 172.21.15.137:0/1510191534 >> 172.21.15.137:0/1510191534 conn(0x7f1e540f09a0 msgr2=0x7f1e540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:04.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 -- 172.21.15.137:0/1510191534 shutdown_connections 2023-11-15T06:01:04.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 -- 172.21.15.137:0/1510191534 wait complete. 2023-11-15T06:01:04.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.885+0000 7f1e53fff700 1 Processor -- start 2023-11-15T06:01:04.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e53fff700 1 -- start start 2023-11-15T06:01:04.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e53fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:04.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e53fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e54060220 con 0x7f1e540f51a0 2023-11-15T06:01:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e52ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e52ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:51828/0 (socket says 172.21.15.137:51828) 2023-11-15T06:01:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e52ffd700 1 -- 172.21.15.137:0/194668426 learned_addr learned my addr 172.21.15.137:0/194668426 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e52ffd700 1 -- 172.21.15.137:0/194668426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44005040 con 0x7f1e540f51a0 2023-11-15T06:01:04.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.886+0000 7f1e52ffd700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1e44013510 tx=0x7f1e4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:04.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e44009320 con 0x7f1e540f51a0 2023-11-15T06:01:04.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e54060420 con 0x7f1e540f51a0 2023-11-15T06:01:04.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e44013f00 con 0x7f1e540f51a0 2023-11-15T06:01:04.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e44014dc0 con 0x7f1e540f51a0 2023-11-15T06:01:04.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e54109880 con 0x7f1e540f51a0 2023-11-15T06:01:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e44014420 con 0x7f1e540f51a0 2023-11-15T06:01:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.887+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54059100 con 0x7f1e540f51a0 2023-11-15T06:01:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.888+0000 7f1e3b7fe700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.888+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e44041110 con 0x7f1e540f51a0 2023-11-15T06:01:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.888+0000 7f1e527fc700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:04.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.888+0000 7f1e527fc700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 0x7f1e3c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1e480096a0 tx=0x7f1e48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:04.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:04.892+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e440095f0 con 0x7f1e540f51a0 2023-11-15T06:01:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.096+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e54045ad0 con 0x7f1e540f51a0 2023-11-15T06:01:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.097+0000 7f1e3b7fe700 1 -- 172.21.15.137:0/194668426 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1e4401cb10 con 0x7f1e540f51a0 2023-11-15T06:01:05.098 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:05.098 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:05.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 msgr2=0x7f1e3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:05.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 0x7f1e3c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1e480096a0 tx=0x7f1e48006b40).stop 2023-11-15T06:01:05.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 msgr2=0x7f1e541091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:05.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1e44013510 tx=0x7f1e4400b040).stop 2023-11-15T06:01:05.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 shutdown_connections 2023-11-15T06:01:05.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1e3c02ff60 0x7f1e3c032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:05.100 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 --2- 172.21.15.137:0/194668426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e540f51a0 0x7f1e541091d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:05.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.098+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 >> 172.21.15.137:0/194668426 conn(0x7f1e540f09a0 msgr2=0x7f1e540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:05.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.099+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 shutdown_connections 2023-11-15T06:01:05.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:05.099+0000 7f1e53fff700 1 -- 172.21.15.137:0/194668426 wait complete. 2023-11-15T06:01:05.101 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:05.025314+0000 mgr.smithi003.cldzke (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:06.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:05.097377+0000 mon.smithi003 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.137:0/194668426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:06.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:06.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:07.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- 172.21.15.137:0/2294187333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 msgr2=0x7f558c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:07.648 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 --2- 172.21.15.137:0/2294187333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f557c009fb0 tx=0x7f557c009b30).stop 2023-11-15T06:01:07.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- 172.21.15.137:0/2294187333 shutdown_connections 2023-11-15T06:01:07.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 --2- 172.21.15.137:0/2294187333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:07.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- 172.21.15.137:0/2294187333 >> 172.21.15.137:0/2294187333 conn(0x7f558c0f09a0 msgr2=0x7f558c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:07.649 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- 172.21.15.137:0/2294187333 shutdown_connections 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- 172.21.15.137:0/2294187333 wait complete. 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 Processor -- start 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.646+0000 7f558b59e700 1 -- start start 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558b59e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558b59e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c0fb5a0 con 0x7f558c0f51a0 2023-11-15T06:01:07.650 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:07.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:51844/0 (socket says 172.21.15.137:51844) 2023-11-15T06:01:07.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558a59c700 1 -- 172.21.15.137:0/3257504562 learned_addr learned my addr 172.21.15.137:0/3257504562 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:07.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558a59c700 1 -- 172.21.15.137:0/3257504562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c005040 con 0x7f558c0f51a0 2023-11-15T06:01:07.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558a59c700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f557c013510 tx=0x7f557c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:07.651 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c009320 con 0x7f558c0f51a0 2023-11-15T06:01:07.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c0fb740 con 0x7f558c0f51a0 2023-11-15T06:01:07.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f557c013f00 con 0x7f558c0f51a0 2023-11-15T06:01:07.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c014dc0 con 0x7f558c0f51a0 2023-11-15T06:01:07.652 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.647+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c0fbbd0 con 0x7f558c0f51a0 2023-11-15T06:01:07.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.648+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c059100 con 0x7f558c0f51a0 2023-11-15T06:01:07.653 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.648+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f557c0095f0 con 0x7f558c0f51a0 2023-11-15T06:01:07.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.648+0000 7f5582ffd700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 0x7f5578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:07.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.649+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f557c041040 con 0x7f558c0f51a0 2023-11-15T06:01:07.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.649+0000 7f5589d9b700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 0x7f5578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:07.654 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.649+0000 7f5589d9b700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 0x7f5578032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55740096a0 tx=0x7f5574006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:07.655 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.653+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f557c014420 con 0x7f558c0f51a0 2023-11-15T06:01:07.861 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.861+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f558c0fabe0 con 0x7f558c0f51a0 2023-11-15T06:01:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.862+0000 7f5582ffd700 1 -- 172.21.15.137:0/3257504562 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f557c010230 con 0x7f558c0f51a0 2023-11-15T06:01:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 msgr2=0x7f5578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 0x7f5578032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55740096a0 tx=0x7f5574006b40).stop 2023-11-15T06:01:07.864 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 msgr2=0x7f558c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f557c013510 tx=0x7f557c00b040).stop 2023-11-15T06:01:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 shutdown_connections 2023-11-15T06:01:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f557802ff60 0x7f5578032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 --2- 172.21.15.137:0/3257504562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f558c0f51a0 0x7f558c0fd120 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:07.865 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 >> 172.21.15.137:0/3257504562 conn(0x7f558c0f09a0 msgr2=0x7f558c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:07.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 shutdown_connections 2023-11-15T06:01:07.866 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:07.863+0000 7f558b59e700 1 -- 172.21.15.137:0/3257504562 wait complete. 2023-11-15T06:01:07.866 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:07.025785+0000 mgr.smithi003.cldzke (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:07.861866+0000 mon.smithi003 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.137:0/3257504562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:09.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:09.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:09.026074+0000 mgr.smithi003.cldzke (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:10.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 -- 172.21.15.137:0/4220702177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 msgr2=0x7f86f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:10.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 --2- 172.21.15.137:0/4220702177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f00f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f86e4009fb0 tx=0x7f86e4009b30).stop 2023-11-15T06:01:10.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 -- 172.21.15.137:0/4220702177 shutdown_connections 2023-11-15T06:01:10.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 --2- 172.21.15.137:0/4220702177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f00f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:10.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 -- 172.21.15.137:0/4220702177 >> 172.21.15.137:0/4220702177 conn(0x7f86f00f1c70 msgr2=0x7f86f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:10.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 -- 172.21.15.137:0/4220702177 shutdown_connections 2023-11-15T06:01:10.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 -- 172.21.15.137:0/4220702177 wait complete. 2023-11-15T06:01:10.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.480+0000 7f86f7191700 1 Processor -- start 2023-11-15T06:01:10.483 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f7191700 1 -- start start 2023-11-15T06:01:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f7191700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f7191700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0069e20 con 0x7f86f00f6510 2023-11-15T06:01:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f4f2d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f4f2d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55042/0 (socket says 172.21.15.137:55042) 2023-11-15T06:01:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f4f2d700 1 -- 172.21.15.137:0/2537287942 learned_addr learned my addr 172.21.15.137:0/2537287942 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:10.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f4f2d700 1 -- 172.21.15.137:0/2537287942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4005040 con 0x7f86f00f6510 2023-11-15T06:01:10.485 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.481+0000 7f86f4f2d700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f86e4013510 tx=0x7f86e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:10.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e4009320 con 0x7f86f00f6510 2023-11-15T06:01:10.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0069fc0 con 0x7f86f00f6510 2023-11-15T06:01:10.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86e4013f00 con 0x7f86f00f6510 2023-11-15T06:01:10.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e4014d90 con 0x7f86f00f6510 2023-11-15T06:01:10.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f006a450 con 0x7f86f00f6510 2023-11-15T06:01:10.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86e40095f0 con 0x7f86f00f6510 2023-11-15T06:01:10.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.482+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f0059100 con 0x7f86f00f6510 2023-11-15T06:01:10.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.483+0000 7f86ed7fa700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 0x7f86e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:10.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.483+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86e40410a0 con 0x7f86f00f6510 2023-11-15T06:01:10.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.483+0000 7f86effff700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 0x7f86e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:10.488 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.483+0000 7f86effff700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 0x7f86e0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f86dc0096a0 tx=0x7f86dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:10.489 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.488+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86e4010230 con 0x7f86f00f6510 2023-11-15T06:01:10.691 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.691+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86f0045ad0 con 0x7f86f00f6510 2023-11-15T06:01:10.692 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.691+0000 7f86ed7fa700 1 -- 172.21.15.137:0/2537287942 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f86e4010230 con 0x7f86f00f6510 2023-11-15T06:01:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:10.693 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:10.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 msgr2=0x7f86e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:10.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 0x7f86e0032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f86dc0096a0 tx=0x7f86dc006b40).stop 2023-11-15T06:01:10.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 msgr2=0x7f86f006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:10.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f86e4013510 tx=0x7f86e400b040).stop 2023-11-15T06:01:10.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 shutdown_connections 2023-11-15T06:01:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f86e002ff60 0x7f86e0032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 --2- 172.21.15.137:0/2537287942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86f00f6510 0x7f86f006b9a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 >> 172.21.15.137:0/2537287942 conn(0x7f86f00f1c70 msgr2=0x7f86f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 shutdown_connections 2023-11-15T06:01:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:10.693+0000 7f86f7191700 1 -- 172.21.15.137:0/2537287942 wait complete. 2023-11-15T06:01:10.696 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:10.691917+0000 mon.smithi003 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.137:0/2537287942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:11.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:11.026440+0000 mgr.smithi003.cldzke (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:12.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:12.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:13.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 -- 172.21.15.137:0/3726018533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 msgr2=0x7f9e640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:13.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/3726018533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e640f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f9e58009fb0 tx=0x7f9e58009b30).stop 2023-11-15T06:01:13.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 -- 172.21.15.137:0/3726018533 shutdown_connections 2023-11-15T06:01:13.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/3726018533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e640f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:13.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 -- 172.21.15.137:0/3726018533 >> 172.21.15.137:0/3726018533 conn(0x7f9e640f1c70 msgr2=0x7f9e640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:13.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 -- 172.21.15.137:0/3726018533 shutdown_connections 2023-11-15T06:01:13.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.295+0000 7f9e6ac24700 1 -- 172.21.15.137:0/3726018533 wait complete. 2023-11-15T06:01:13.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.296+0000 7f9e6ac24700 1 Processor -- start 2023-11-15T06:01:13.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.296+0000 7f9e6ac24700 1 -- start start 2023-11-15T06:01:13.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.296+0000 7f9e6ac24700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:13.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.296+0000 7f9e6ac24700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64106c20 con 0x7f9e640f6510 2023-11-15T06:01:13.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.296+0000 7f9e689c0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:13.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e689c0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55054/0 (socket says 172.21.15.137:55054) 2023-11-15T06:01:13.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e689c0700 1 -- 172.21.15.137:0/2238633364 learned_addr learned my addr 172.21.15.137:0/2238633364 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:13.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e689c0700 1 -- 172.21.15.137:0/2238633364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e58005040 con 0x7f9e640f6510 2023-11-15T06:01:13.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e689c0700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9e58013510 tx=0x7f9e5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:13.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e58009320 con 0x7f9e640f6510 2023-11-15T06:01:13.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e58013f00 con 0x7f9e640f6510 2023-11-15T06:01:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64106e20 con 0x7f9e640f6510 2023-11-15T06:01:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e58014d90 con 0x7f9e640f6510 2023-11-15T06:01:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.297+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e641072b0 con 0x7f9e640f6510 2023-11-15T06:01:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.298+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e580263a0 con 0x7f9e640f6510 2023-11-15T06:01:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.299+0000 7f9e617fa700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 0x7f9e54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:13.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.299+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e58041130 con 0x7f9e640f6510 2023-11-15T06:01:13.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.299+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64059100 con 0x7f9e640f6510 2023-11-15T06:01:13.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.299+0000 7f9e63fff700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 0x7f9e54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:13.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.299+0000 7f9e63fff700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 0x7f9e54032470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:13.306 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.304+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e58012920 con 0x7f9e640f6510 2023-11-15T06:01:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.509+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e64045ad0 con 0x7f9e640f6510 2023-11-15T06:01:13.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.510+0000 7f9e617fa700 1 -- 172.21.15.137:0/2238633364 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9e5802b020 con 0x7f9e640f6510 2023-11-15T06:01:13.512 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:13.512 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 msgr2=0x7f9e54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 0x7f9e54032470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).stop 2023-11-15T06:01:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 msgr2=0x7f9e64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:13.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9e58013510 tx=0x7f9e5800b040).stop 2023-11-15T06:01:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 shutdown_connections 2023-11-15T06:01:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9e5402ffb0 0x7f9e54032470 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 --2- 172.21.15.137:0/2238633364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9e640f6510 0x7f9e64106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 >> 172.21.15.137:0/2238633364 conn(0x7f9e640f1c70 msgr2=0x7f9e640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:13.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 shutdown_connections 2023-11-15T06:01:13.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:13.511+0000 7f9e6ac24700 1 -- 172.21.15.137:0/2238633364 wait complete. 2023-11-15T06:01:13.515 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:14.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:13.026887+0000 mgr.smithi003.cldzke (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:14.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:13.510397+0000 mon.smithi003 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.137:0/2238633364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:14.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:14.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:16.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 -- 172.21.15.137:0/4168045751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc80f6510 msgr2=0x7fcbc80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:16.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/4168045751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc80f6510 0x7fcbc80f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fcbb8009fb0 tx=0x7fcbb8009b30).stop 2023-11-15T06:01:16.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 -- 172.21.15.137:0/4168045751 shutdown_connections 2023-11-15T06:01:16.043 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/4168045751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc80f6510 0x7fcbc80f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:16.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 -- 172.21.15.137:0/4168045751 >> 172.21.15.137:0/4168045751 conn(0x7fcbc80f1c70 msgr2=0x7fcbc80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:16.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 -- 172.21.15.137:0/4168045751 shutdown_connections 2023-11-15T06:01:16.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.040+0000 7fcbcff6b700 1 -- 172.21.15.137:0/4168045751 wait complete. 2023-11-15T06:01:16.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcff6b700 1 Processor -- start 2023-11-15T06:01:16.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcff6b700 1 -- start start 2023-11-15T06:01:16.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcff6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:16.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcff6b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc8107150 con 0x7fcbc8106840 2023-11-15T06:01:16.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcdd07700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:16.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcdd07700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55066/0 (socket says 172.21.15.137:55066) 2023-11-15T06:01:16.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcdd07700 1 -- 172.21.15.137:0/588226715 learned_addr learned my addr 172.21.15.137:0/588226715 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:16.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcdd07700 1 -- 172.21.15.137:0/588226715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8005040 con 0x7fcbc8106840 2023-11-15T06:01:16.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.041+0000 7fcbcdd07700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fcbb8009f80 tx=0x7fcbb8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:16.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.042+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb80136c0 con 0x7fcbc8106840 2023-11-15T06:01:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.042+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc8107350 con 0x7fcbc8106840 2023-11-15T06:01:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.042+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbb80095f0 con 0x7fcbc8106840 2023-11-15T06:01:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.042+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb8014420 con 0x7fcbc8106840 2023-11-15T06:01:16.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.042+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc8107ea0 con 0x7fcbc8106840 2023-11-15T06:01:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.043+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc8059100 con 0x7fcbc8106840 2023-11-15T06:01:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.043+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbb8025060 con 0x7fcbc8106840 2023-11-15T06:01:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.043+0000 7fcbbe7fc700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 0x7fcbb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.043+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbb8040da0 con 0x7fcbc8106840 2023-11-15T06:01:16.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.043+0000 7fcbcd506700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 0x7fcbb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.044+0000 7fcbcd506700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 0x7fcbb4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fcbc40096a0 tx=0x7fcbc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:16.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.047+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbb801cb60 con 0x7fcbc8106840 2023-11-15T06:01:16.252 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.252+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbc8045ad0 con 0x7fcbc8106840 2023-11-15T06:01:16.253 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.252+0000 7fcbbe7fc700 1 -- 172.21.15.137:0/588226715 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcbb800f280 con 0x7fcbc8106840 2023-11-15T06:01:16.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:16.253 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 msgr2=0x7fcbb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 0x7fcbb4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fcbc40096a0 tx=0x7fcbc4006b40).stop 2023-11-15T06:01:16.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 msgr2=0x7fcbc8106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fcbb8009f80 tx=0x7fcbb8009b30).stop 2023-11-15T06:01:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 shutdown_connections 2023-11-15T06:01:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcbb402ff60 0x7fcbb4032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:16.256 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 --2- 172.21.15.137:0/588226715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc8106840 0x7fcbc8106c60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 >> 172.21.15.137:0/588226715 conn(0x7fcbc80f1c70 msgr2=0x7fcbc80f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 shutdown_connections 2023-11-15T06:01:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:16.254+0000 7fcbcff6b700 1 -- 172.21.15.137:0/588226715 wait complete. 2023-11-15T06:01:16.257 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:15.027143+0000 mgr.smithi003.cldzke (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:17.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:17 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:16.252682+0000 mon.smithi003 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.137:0/588226715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:17.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:17.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:17.027411+0000 mgr.smithi003.cldzke (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:18.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1389669243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 msgr2=0x7fe2e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:18.872 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1389669243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e80f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fe2dc009fb0 tx=0x7fe2dc009b30).stop 2023-11-15T06:01:18.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1389669243 shutdown_connections 2023-11-15T06:01:18.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1389669243 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e80f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:18.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1389669243 >> 172.21.15.137:0/1389669243 conn(0x7fe2e80f1c70 msgr2=0x7fe2e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:18.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1389669243 shutdown_connections 2023-11-15T06:01:18.873 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.870+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1389669243 wait complete. 2023-11-15T06:01:18.874 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.871+0000 7fe2ede3c700 1 Processor -- start 2023-11-15T06:01:18.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.871+0000 7fe2ede3c700 1 -- start start 2023-11-15T06:01:18.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.871+0000 7fe2ede3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:18.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.871+0000 7fe2ede3c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8106c20 con 0x7fe2e80f6510 2023-11-15T06:01:18.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2e77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:18.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2e77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:55074/0 (socket says 172.21.15.137:55074) 2023-11-15T06:01:18.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2e77fe700 1 -- 172.21.15.137:0/1957364288 learned_addr learned my addr 172.21.15.137:0/1957364288 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:18.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2e77fe700 1 -- 172.21.15.137:0/1957364288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc005040 con 0x7fe2e80f6510 2023-11-15T06:01:18.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2e77fe700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fe2dc013510 tx=0x7fe2dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:18.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc009320 con 0x7fe2e80f6510 2023-11-15T06:01:18.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e8106e20 con 0x7fe2e80f6510 2023-11-15T06:01:18.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2dc013f00 con 0x7fe2e80f6510 2023-11-15T06:01:18.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2dc014d90 con 0x7fe2e80f6510 2023-11-15T06:01:18.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.872+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e81072b0 con 0x7fe2e80f6510 2023-11-15T06:01:18.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.873+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2dc0263a0 con 0x7fe2e80f6510 2023-11-15T06:01:18.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.873+0000 7fe2cffff700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 0x7fe2d0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:18.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.873+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2dc040fa0 con 0x7fe2e80f6510 2023-11-15T06:01:18.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.873+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8059100 con 0x7fe2e80f6510 2023-11-15T06:01:18.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.874+0000 7fe2e6ffd700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 0x7fe2d0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:18.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.874+0000 7fe2e6ffd700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 0x7fe2d0032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe2d80096a0 tx=0x7fe2d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:18.880 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:18.878+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2dc012920 con 0x7fe2e80f6510 2023-11-15T06:01:19.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.081+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2e8045ad0 con 0x7fe2e80f6510 2023-11-15T06:01:19.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.081+0000 7fe2cffff700 1 -- 172.21.15.137:0/1957364288 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe2dc02b020 con 0x7fe2e80f6510 2023-11-15T06:01:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:19.083 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:19.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 msgr2=0x7fe2d0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:19.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 0x7fe2d0032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe2d80096a0 tx=0x7fe2d8006b40).stop 2023-11-15T06:01:19.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 msgr2=0x7fe2e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:19.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fe2dc013510 tx=0x7fe2dc00b040).stop 2023-11-15T06:01:19.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 shutdown_connections 2023-11-15T06:01:19.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe2d002ffb0 0x7fe2d0032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:19.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 --2- 172.21.15.137:0/1957364288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2e80f6510 0x7fe2e8106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:19.086 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 >> 172.21.15.137:0/1957364288 conn(0x7fe2e80f1c70 msgr2=0x7fe2e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:19.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 shutdown_connections 2023-11-15T06:01:19.087 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:19.083+0000 7fe2ede3c700 1 -- 172.21.15.137:0/1957364288 wait complete. 2023-11-15T06:01:19.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:19.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:19 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:19.082786+0000 mon.smithi003 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.137:0/1957364288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:20.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:19.027797+0000 mgr.smithi003.cldzke (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:20.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:20.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:21.694 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 -- 172.21.15.137:0/1140735212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 msgr2=0x7fa5500f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:21.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 --2- 172.21.15.137:0/1140735212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5500f9150 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fa540009fb0 tx=0x7fa540009b30).stop 2023-11-15T06:01:21.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 -- 172.21.15.137:0/1140735212 shutdown_connections 2023-11-15T06:01:21.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 --2- 172.21.15.137:0/1140735212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5500f9150 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:21.695 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 -- 172.21.15.137:0/1140735212 >> 172.21.15.137:0/1140735212 conn(0x7fa5500f0b70 msgr2=0x7fa5500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 -- 172.21.15.137:0/1140735212 shutdown_connections 2023-11-15T06:01:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 -- 172.21.15.137:0/1140735212 wait complete. 2023-11-15T06:01:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.692+0000 7fa5566e5700 1 Processor -- start 2023-11-15T06:01:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa5566e5700 1 -- start start 2023-11-15T06:01:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa5566e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:21.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa5566e5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa550106bd0 con 0x7fa550060220 2023-11-15T06:01:21.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:21.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33512/0 (socket says 172.21.15.137:33512) 2023-11-15T06:01:21.697 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.693+0000 7fa54ffff700 1 -- 172.21.15.137:0/3714935767 learned_addr learned my addr 172.21.15.137:0/3714935767 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:21.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa54ffff700 1 -- 172.21.15.137:0/3714935767 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540005040 con 0x7fa550060220 2023-11-15T06:01:21.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa54ffff700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa540013510 tx=0x7fa54000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:21.699 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa540009320 con 0x7fa550060220 2023-11-15T06:01:21.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa550106dd0 con 0x7fa550060220 2023-11-15T06:01:21.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa540013f00 con 0x7fa550060220 2023-11-15T06:01:21.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa540014dc0 con 0x7fa550060220 2023-11-15T06:01:21.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.694+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa550107260 con 0x7fa550060220 2023-11-15T06:01:21.700 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.695+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa54001d3d0 con 0x7fa550060220 2023-11-15T06:01:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.695+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa550045ad0 con 0x7fa550060220 2023-11-15T06:01:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.695+0000 7fa54cff9700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 0x7fa53c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.695+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5400413c0 con 0x7fa550060220 2023-11-15T06:01:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.695+0000 7fa54f7fe700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 0x7fa53c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.696+0000 7fa54f7fe700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 0x7fa53c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa5380096a0 tx=0x7fa538006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:21.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.700+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa54001ccb0 con 0x7fa550060220 2023-11-15T06:01:21.904 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.902+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa550108110 con 0x7fa550060220 2023-11-15T06:01:21.905 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.903+0000 7fa54cff9700 1 -- 172.21.15.137:0/3714935767 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa54000f340 con 0x7fa550060220 2023-11-15T06:01:21.906 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:21.907 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:21.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.904+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 msgr2=0x7fa53c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:21.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.904+0000 7fa5566e5700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 0x7fa53c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa5380096a0 tx=0x7fa538006b40).stop 2023-11-15T06:01:21.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.904+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 msgr2=0x7fa5501066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:21.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.904+0000 7fa5566e5700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa540013510 tx=0x7fa54000b040).stop 2023-11-15T06:01:21.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 shutdown_connections 2023-11-15T06:01:21.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa53c02ff60 0x7fa53c032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:21.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 --2- 172.21.15.137:0/3714935767 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa550060220 0x7fa5501066e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:21.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 >> 172.21.15.137:0/3714935767 conn(0x7fa5500f0b70 msgr2=0x7fa5500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:21.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 shutdown_connections 2023-11-15T06:01:21.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:21.905+0000 7fa5566e5700 1 -- 172.21.15.137:0/3714935767 wait complete. 2023-11-15T06:01:21.913 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:21.028169+0000 mgr.smithi003.cldzke (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:22.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:21.904376+0000 mon.smithi003 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.137:0/3714935767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:23.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:23.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:23.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:23 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:23.028524+0000 mgr.smithi003.cldzke (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:24.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 -- 172.21.15.137:0/2089783485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 msgr2=0x7fd8100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:24.486 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2089783485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd8100f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fd80c009fb0 tx=0x7fd80c009b30).stop 2023-11-15T06:01:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 -- 172.21.15.137:0/2089783485 shutdown_connections 2023-11-15T06:01:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2089783485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd8100f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 -- 172.21.15.137:0/2089783485 >> 172.21.15.137:0/2089783485 conn(0x7fd8100f1c70 msgr2=0x7fd8100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 -- 172.21.15.137:0/2089783485 shutdown_connections 2023-11-15T06:01:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.484+0000 7fd817f0a700 1 -- 172.21.15.137:0/2089783485 wait complete. 2023-11-15T06:01:24.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.485+0000 7fd817f0a700 1 Processor -- start 2023-11-15T06:01:24.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.485+0000 7fd817f0a700 1 -- start start 2023-11-15T06:01:24.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.485+0000 7fd817f0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:24.490 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.485+0000 7fd817f0a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810106c20 con 0x7fd8100f6510 2023-11-15T06:01:24.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd815ca6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:24.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd815ca6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33544/0 (socket says 172.21.15.137:33544) 2023-11-15T06:01:24.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd815ca6700 1 -- 172.21.15.137:0/2506552700 learned_addr learned my addr 172.21.15.137:0/2506552700 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:24.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd815ca6700 1 -- 172.21.15.137:0/2506552700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c005040 con 0x7fd8100f6510 2023-11-15T06:01:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd815ca6700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd80c013510 tx=0x7fd80c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c009320 con 0x7fd8100f6510 2023-11-15T06:01:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd810106e20 con 0x7fd8100f6510 2023-11-15T06:01:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd80c013f00 con 0x7fd8100f6510 2023-11-15T06:01:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80c014d90 con 0x7fd8100f6510 2023-11-15T06:01:24.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.486+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101072b0 con 0x7fd8100f6510 2023-11-15T06:01:24.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.487+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd80c0263a0 con 0x7fd8100f6510 2023-11-15T06:01:24.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.487+0000 7fd8067fc700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 0x7fd7fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:24.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.487+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd80c0410b0 con 0x7fd8100f6510 2023-11-15T06:01:24.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.487+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd810045ad0 con 0x7fd8100f6510 2023-11-15T06:01:24.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.487+0000 7fd8154a5700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 0x7fd7fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:24.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.488+0000 7fd8154a5700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 0x7fd7fc032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd8000096a0 tx=0x7fd800006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:24.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.492+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd80c012920 con 0x7fd8100f6510 2023-11-15T06:01:24.701 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.700+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd810107fd0 con 0x7fd8100f6510 2023-11-15T06:01:24.702 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.701+0000 7fd8067fc700 1 -- 172.21.15.137:0/2506552700 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd80c02b020 con 0x7fd8100f6510 2023-11-15T06:01:24.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:24.703 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:24.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 msgr2=0x7fd7fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:24.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 0x7fd7fc032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd8000096a0 tx=0x7fd800006b40).stop 2023-11-15T06:01:24.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 msgr2=0x7fd810106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:24.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fd80c013510 tx=0x7fd80c00b040).stop 2023-11-15T06:01:24.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 shutdown_connections 2023-11-15T06:01:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd7fc02ff60 0x7fd7fc032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 --2- 172.21.15.137:0/2506552700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd8100f6510 0x7fd810106730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 >> 172.21.15.137:0/2506552700 conn(0x7fd8100f1c70 msgr2=0x7fd8100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 shutdown_connections 2023-11-15T06:01:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:24.702+0000 7fd817f0a700 1 -- 172.21.15.137:0/2506552700 wait complete. 2023-11-15T06:01:24.706 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:25.180 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:24.701998+0000 mon.smithi003 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.137:0/2506552700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:26.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:26.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:26.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:25.028862+0000 mgr.smithi003.cldzke (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:27.316 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 -- 172.21.15.137:0/3147883667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 msgr2=0x7f68180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:27.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 --2- 172.21.15.137:0/3147883667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f68180f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f680c009fb0 tx=0x7f680c009b30).stop 2023-11-15T06:01:27.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 -- 172.21.15.137:0/3147883667 shutdown_connections 2023-11-15T06:01:27.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 --2- 172.21.15.137:0/3147883667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f68180f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:27.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 -- 172.21.15.137:0/3147883667 >> 172.21.15.137:0/3147883667 conn(0x7f68180f1c70 msgr2=0x7f68180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:27.317 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 -- 172.21.15.137:0/3147883667 shutdown_connections 2023-11-15T06:01:27.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 -- 172.21.15.137:0/3147883667 wait complete. 2023-11-15T06:01:27.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.314+0000 7f681db7f700 1 Processor -- start 2023-11-15T06:01:27.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f681db7f700 1 -- start start 2023-11-15T06:01:27.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f681db7f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:27.318 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f681db7f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6818069e20 con 0x7f68180f6510 2023-11-15T06:01:27.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f68177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:27.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f68177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33562/0 (socket says 172.21.15.137:33562) 2023-11-15T06:01:27.319 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f68177fe700 1 -- 172.21.15.137:0/1889709304 learned_addr learned my addr 172.21.15.137:0/1889709304 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:27.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f68177fe700 1 -- 172.21.15.137:0/1889709304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c005040 con 0x7f68180f6510 2023-11-15T06:01:27.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.315+0000 7f68177fe700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f680c013510 tx=0x7f680c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:27.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c009320 con 0x7f68180f6510 2023-11-15T06:01:27.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6818069fc0 con 0x7f68180f6510 2023-11-15T06:01:27.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f680c013f00 con 0x7f68180f6510 2023-11-15T06:01:27.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c014d90 con 0x7f68180f6510 2023-11-15T06:01:27.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681806a450 con 0x7f68180f6510 2023-11-15T06:01:27.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f680c0095f0 con 0x7f68180f6510 2023-11-15T06:01:27.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.316+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6818059100 con 0x7f68180f6510 2023-11-15T06:01:27.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.317+0000 7f67fffff700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 0x7f6800032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:27.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.317+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f680c0410c0 con 0x7f68180f6510 2023-11-15T06:01:27.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.317+0000 7f6816ffd700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 0x7f6800032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:27.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.317+0000 7f6816ffd700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 0x7f6800032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f68080096a0 tx=0x7f6808006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:27.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.322+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f680c014420 con 0x7f68180f6510 2023-11-15T06:01:27.528 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.527+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68180fbf60 con 0x7f68180f6510 2023-11-15T06:01:27.529 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.528+0000 7f67fffff700 1 -- 172.21.15.137:0/1889709304 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f680c010230 con 0x7f68180f6510 2023-11-15T06:01:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:27.530 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:27.531 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 msgr2=0x7f6800032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:27.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 0x7f6800032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f68080096a0 tx=0x7f6808006b40).stop 2023-11-15T06:01:27.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 msgr2=0x7f681806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:27.532 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f680c013510 tx=0x7f680c00b040).stop 2023-11-15T06:01:27.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 shutdown_connections 2023-11-15T06:01:27.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680002ff60 0x7f6800032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:27.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 --2- 172.21.15.137:0/1889709304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68180f6510 0x7f681806b9a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:27.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 >> 172.21.15.137:0/1889709304 conn(0x7f68180f1c70 msgr2=0x7f68180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:27.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 shutdown_connections 2023-11-15T06:01:27.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:27.529+0000 7f681db7f700 1 -- 172.21.15.137:0/1889709304 wait complete. 2023-11-15T06:01:27.535 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:28.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:27.029204+0000 mgr.smithi003.cldzke (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:27.529083+0000 mon.smithi003 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.137:0/1889709304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:28.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:28.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:30.112 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 -- 172.21.15.137:0/317004472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 msgr2=0x7f50b80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:30.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 --2- 172.21.15.137:0/317004472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f8f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f50b4009fb0 tx=0x7f50b4009b30).stop 2023-11-15T06:01:30.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 -- 172.21.15.137:0/317004472 shutdown_connections 2023-11-15T06:01:30.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 --2- 172.21.15.137:0/317004472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f8f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:30.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 -- 172.21.15.137:0/317004472 >> 172.21.15.137:0/317004472 conn(0x7f50b80f09a0 msgr2=0x7f50b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:30.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 -- 172.21.15.137:0/317004472 shutdown_connections 2023-11-15T06:01:30.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.110+0000 7f50bf549700 1 -- 172.21.15.137:0/317004472 wait complete. 2023-11-15T06:01:30.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50bf549700 1 Processor -- start 2023-11-15T06:01:30.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50bf549700 1 -- start start 2023-11-15T06:01:30.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50bf549700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:30.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50bf549700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80f4e20 con 0x7f50b80601e0 2023-11-15T06:01:30.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50be547700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:30.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50be547700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39032/0 (socket says 172.21.15.137:39032) 2023-11-15T06:01:30.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50be547700 1 -- 172.21.15.137:0/3150510945 learned_addr learned my addr 172.21.15.137:0/3150510945 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:30.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.111+0000 7f50be547700 1 -- 172.21.15.137:0/3150510945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4005040 con 0x7f50b80601e0 2023-11-15T06:01:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50be547700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f50b4013510 tx=0x7f50b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b4009320 con 0x7f50b80601e0 2023-11-15T06:01:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b80f4fc0 con 0x7f50b80601e0 2023-11-15T06:01:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50b4013f00 con 0x7f50b80601e0 2023-11-15T06:01:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b4014d90 con 0x7f50b80601e0 2023-11-15T06:01:30.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.112+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b80f5450 con 0x7f50b80601e0 2023-11-15T06:01:30.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.113+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50b4014420 con 0x7f50b80601e0 2023-11-15T06:01:30.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.113+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b8059100 con 0x7f50b80601e0 2023-11-15T06:01:30.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.113+0000 7f50aeffd700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:30.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.113+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50b40410c0 con 0x7f50b80601e0 2023-11-15T06:01:30.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.113+0000 7f50bdd46700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:30.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.114+0000 7f50bdd46700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 0x7f50a4032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f50a80096a0 tx=0x7f50a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:30.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.118+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50b40095f0 con 0x7f50b80601e0 2023-11-15T06:01:30.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.321+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50b80fbcd0 con 0x7f50b80601e0 2023-11-15T06:01:30.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.322+0000 7f50aeffd700 1 -- 172.21.15.137:0/3150510945 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f50b401db70 con 0x7f50b80601e0 2023-11-15T06:01:30.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:30.324 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:30.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 msgr2=0x7f50a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:30.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 0x7f50a4032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f50a80096a0 tx=0x7f50a8006b40).stop 2023-11-15T06:01:30.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 msgr2=0x7f50b80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f50b4013510 tx=0x7f50b400b040).stop 2023-11-15T06:01:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 shutdown_connections 2023-11-15T06:01:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 --2- 172.21.15.137:0/3150510945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50b80601e0 0x7f50b80f69a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.323+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 >> 172.21.15.137:0/3150510945 conn(0x7f50b80f09a0 msgr2=0x7f50b80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:30.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.324+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 shutdown_connections 2023-11-15T06:01:30.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:30.324+0000 7f50bf549700 1 -- 172.21.15.137:0/3150510945 wait complete. 2023-11-15T06:01:30.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:29.029534+0000 mgr.smithi003.cldzke (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:31.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:30.323449+0000 mon.smithi003 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.137:0/3150510945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:31.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:31.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:32.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:31.029875+0000 mgr.smithi003.cldzke (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:32.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 -- 172.21.15.137:0/3282456122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 msgr2=0x7f681c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:32.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 --2- 172.21.15.137:0/3282456122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c0f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f680c009fb0 tx=0x7f680c009b30).stop 2023-11-15T06:01:32.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 -- 172.21.15.137:0/3282456122 shutdown_connections 2023-11-15T06:01:32.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 --2- 172.21.15.137:0/3282456122 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c0f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:32.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 -- 172.21.15.137:0/3282456122 >> 172.21.15.137:0/3282456122 conn(0x7f681c0f0b70 msgr2=0x7f681c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:32.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 -- 172.21.15.137:0/3282456122 shutdown_connections 2023-11-15T06:01:32.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.885+0000 7f6823422700 1 -- 172.21.15.137:0/3282456122 wait complete. 2023-11-15T06:01:32.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f6823422700 1 Processor -- start 2023-11-15T06:01:32.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f6823422700 1 -- start start 2023-11-15T06:01:32.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f6823422700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:32.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f6823422700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c108e10 con 0x7f681c0f5410 2023-11-15T06:01:32.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f68211be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:32.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f68211be700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39060/0 (socket says 172.21.15.137:39060) 2023-11-15T06:01:32.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f68211be700 1 -- 172.21.15.137:0/2836373780 learned_addr learned my addr 172.21.15.137:0/2836373780 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:32.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f68211be700 1 -- 172.21.15.137:0/2836373780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c005040 con 0x7f681c0f5410 2023-11-15T06:01:32.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.886+0000 7f68211be700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f680c013510 tx=0x7f680c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:32.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.887+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c009320 con 0x7f681c0f5410 2023-11-15T06:01:32.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.887+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c109010 con 0x7f681c0f5410 2023-11-15T06:01:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.887+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f680c013f00 con 0x7f681c0f5410 2023-11-15T06:01:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.887+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f680c014d90 con 0x7f681c0f5410 2023-11-15T06:01:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.887+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1094a0 con 0x7f681c0f5410 2023-11-15T06:01:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.888+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f680c0095f0 con 0x7f681c0f5410 2023-11-15T06:01:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.888+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c045ad0 con 0x7f681c0f5410 2023-11-15T06:01:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.888+0000 7f6811ffb700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 0x7f6808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.888+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f680c041000 con 0x7f681c0f5410 2023-11-15T06:01:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.888+0000 7f68209bd700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 0x7f6808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.889+0000 7f68209bd700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 0x7f6808032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f68180096a0 tx=0x7f6818006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:32.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:32.893+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f680c01d7f0 con 0x7f681c0f5410 2023-11-15T06:01:33.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.097+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f681c10a350 con 0x7f681c0f5410 2023-11-15T06:01:33.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.098+0000 7f6811ffb700 1 -- 172.21.15.137:0/2836373780 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f680c010230 con 0x7f681c0f5410 2023-11-15T06:01:33.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:33.100 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:33.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.099+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 msgr2=0x7f6808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:33.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.099+0000 7f6823422700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 0x7f6808032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f68180096a0 tx=0x7f6818006b40).stop 2023-11-15T06:01:33.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.099+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 msgr2=0x7f681c108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:33.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.099+0000 7f6823422700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f680c013510 tx=0x7f680c00b040).stop 2023-11-15T06:01:33.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.099+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 shutdown_connections 2023-11-15T06:01:33.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.100+0000 7f6823422700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f680802ff60 0x7f6808032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:33.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.100+0000 7f6823422700 1 --2- 172.21.15.137:0/2836373780 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f681c0f5410 0x7f681c108920 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:33.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.100+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 >> 172.21.15.137:0/2836373780 conn(0x7f681c0f0b70 msgr2=0x7f681c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:33.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.100+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 shutdown_connections 2023-11-15T06:01:33.104 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:33.100+0000 7f6823422700 1 -- 172.21.15.137:0/2836373780 wait complete. 2023-11-15T06:01:33.104 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:33.030212+0000 mgr.smithi003.cldzke (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:34.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:33.099277+0000 mon.smithi003 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.137:0/2836373780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:34.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:34.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:35.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 -- 172.21.15.137:0/3038101249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 msgr2=0x7fdb740f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:35.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/3038101249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb740f6850 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fdb68009fb0 tx=0x7fdb68009b30).stop 2023-11-15T06:01:35.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 -- 172.21.15.137:0/3038101249 shutdown_connections 2023-11-15T06:01:35.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/3038101249 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb740f6850 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:35.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 -- 172.21.15.137:0/3038101249 >> 172.21.15.137:0/3038101249 conn(0x7fdb740f1b90 msgr2=0x7fdb740f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:35.626 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 -- 172.21.15.137:0/3038101249 shutdown_connections 2023-11-15T06:01:35.627 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.623+0000 7fdb7bf39700 1 -- 172.21.15.137:0/3038101249 wait complete. 2023-11-15T06:01:35.628 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.624+0000 7fdb7bf39700 1 Processor -- start 2023-11-15T06:01:35.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.624+0000 7fdb7bf39700 1 -- start start 2023-11-15T06:01:35.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.624+0000 7fdb7bf39700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:35.629 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.624+0000 7fdb7bf39700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb74106ec0 con 0x7fdb740f6430 2023-11-15T06:01:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb79cd5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb79cd5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39078/0 (socket says 172.21.15.137:39078) 2023-11-15T06:01:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb79cd5700 1 -- 172.21.15.137:0/2461129953 learned_addr learned my addr 172.21.15.137:0/2461129953 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb79cd5700 1 -- 172.21.15.137:0/2461129953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb68005040 con 0x7fdb740f6430 2023-11-15T06:01:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb79cd5700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdb68013510 tx=0x7fdb6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:35.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb68009320 con 0x7fdb740f6430 2023-11-15T06:01:35.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb741070c0 con 0x7fdb740f6430 2023-11-15T06:01:35.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb68013f00 con 0x7fdb740f6430 2023-11-15T06:01:35.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb68014d90 con 0x7fdb740f6430 2023-11-15T06:01:35.631 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.625+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb74107550 con 0x7fdb740f6430 2023-11-15T06:01:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.626+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb74045ad0 con 0x7fdb740f6430 2023-11-15T06:01:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.626+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb680095f0 con 0x7fdb740f6430 2023-11-15T06:01:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.626+0000 7fdb667fc700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.626+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb68041000 con 0x7fdb740f6430 2023-11-15T06:01:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.627+0000 7fdb794d4700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.627+0000 7fdb794d4700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 0x7fdb60032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdb700096a0 tx=0x7fdb70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.631+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb68014420 con 0x7fdb740f6430 2023-11-15T06:01:35.842 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.841+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb7405ea50 con 0x7fdb740f6430 2023-11-15T06:01:35.843 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.841+0000 7fdb667fc700 1 -- 172.21.15.137:0/2461129953 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fdb68010230 con 0x7fdb740f6430 2023-11-15T06:01:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:35.843 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:35.844 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 msgr2=0x7fdb60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:35.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 0x7fdb60032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdb700096a0 tx=0x7fdb70006b40).stop 2023-11-15T06:01:35.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 msgr2=0x7fdb741069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:35.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdb68013510 tx=0x7fdb6800b040).stop 2023-11-15T06:01:35.845 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 shutdown_connections 2023-11-15T06:01:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 --2- 172.21.15.137:0/2461129953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdb740f6430 0x7fdb741069d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 >> 172.21.15.137:0/2461129953 conn(0x7fdb740f1b90 msgr2=0x7fdb740f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 shutdown_connections 2023-11-15T06:01:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:35.843+0000 7fdb7bf39700 1 -- 172.21.15.137:0/2461129953 wait complete. 2023-11-15T06:01:35.847 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:36.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:35.030493+0000 mgr.smithi003.cldzke (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:35.116753+0000 mon.smithi003 (mon.0) 278 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:01:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:35.119396+0000 mon.smithi003 (mon.0) 279 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T06:01:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:35.842741+0000 mon.smithi003 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.137:0/2461129953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:37.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:37.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:38.410 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 -- 172.21.15.137:0/41689891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 msgr2=0x7fd2f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:38.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/41689891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fd2e0009fb0 tx=0x7fd2e0009b30).stop 2023-11-15T06:01:38.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 -- 172.21.15.137:0/41689891 shutdown_connections 2023-11-15T06:01:38.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/41689891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:38.411 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 -- 172.21.15.137:0/41689891 >> 172.21.15.137:0/41689891 conn(0x7fd2f00f0b70 msgr2=0x7fd2f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:38.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 -- 172.21.15.137:0/41689891 shutdown_connections 2023-11-15T06:01:38.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.407+0000 7fd2f79bf700 1 -- 172.21.15.137:0/41689891 wait complete. 2023-11-15T06:01:38.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f79bf700 1 Processor -- start 2023-11-15T06:01:38.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f79bf700 1 -- start start 2023-11-15T06:01:38.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f79bf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:38.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f79bf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00fb820 con 0x7fd2f00f5410 2023-11-15T06:01:38.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f575b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:38.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f575b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39092/0 (socket says 172.21.15.137:39092) 2023-11-15T06:01:38.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f575b700 1 -- 172.21.15.137:0/2169216564 learned_addr learned my addr 172.21.15.137:0/2169216564 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:38.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.408+0000 7fd2f575b700 1 -- 172.21.15.137:0/2169216564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0005040 con 0x7fd2f00f5410 2023-11-15T06:01:38.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2f575b700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd2e0013510 tx=0x7fd2e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:38.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2e0009320 con 0x7fd2f00f5410 2023-11-15T06:01:38.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f00fb9c0 con 0x7fd2f00f5410 2023-11-15T06:01:38.414 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2e0013f00 con 0x7fd2f00f5410 2023-11-15T06:01:38.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2e0014d90 con 0x7fd2f00f5410 2023-11-15T06:01:38.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.409+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f00fbe50 con 0x7fd2f00f5410 2023-11-15T06:01:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.410+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0059100 con 0x7fd2f00f5410 2023-11-15T06:01:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.410+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2e0014420 con 0x7fd2f00f5410 2023-11-15T06:01:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.410+0000 7fd2e5ffb700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 0x7fd2dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.410+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2e00410d0 con 0x7fd2f00f5410 2023-11-15T06:01:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.410+0000 7fd2f4f5a700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 0x7fd2dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:38.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.411+0000 7fd2f4f5a700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 0x7fd2dc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd2ec0096a0 tx=0x7fd2ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:38.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.415+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2e00095f0 con 0x7fd2f00f5410 2023-11-15T06:01:38.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:37.030838+0000 mgr.smithi003.cldzke (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:38.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.619+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2f0045ad0 con 0x7fd2f00f5410 2023-11-15T06:01:38.621 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.620+0000 7fd2e5ffb700 1 -- 172.21.15.137:0/2169216564 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd2e001db70 con 0x7fd2f00f5410 2023-11-15T06:01:38.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:38.622 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:38.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.621+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 msgr2=0x7fd2dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:38.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.621+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 0x7fd2dc032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd2ec0096a0 tx=0x7fd2ec006b40).stop 2023-11-15T06:01:38.623 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.621+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 msgr2=0x7fd2f00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:38.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.621+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd2e0013510 tx=0x7fd2e000b040).stop 2023-11-15T06:01:38.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.621+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 shutdown_connections 2023-11-15T06:01:38.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.622+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd2dc02ff60 0x7fd2dc032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:38.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.622+0000 7fd2f79bf700 1 --2- 172.21.15.137:0/2169216564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2f00f5410 0x7fd2f00fd3a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:38.624 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.622+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 >> 172.21.15.137:0/2169216564 conn(0x7fd2f00f0b70 msgr2=0x7fd2f005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:38.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.622+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 shutdown_connections 2023-11-15T06:01:38.625 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:38.622+0000 7fd2f79bf700 1 -- 172.21.15.137:0/2169216564 wait complete. 2023-11-15T06:01:38.625 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:39.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:39 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:38.621335+0000 mon.smithi003 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.137:0/2169216564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:40.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:40.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:39.031069+0000 mgr.smithi003.cldzke (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:41.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 -- 172.21.15.137:0/2247029107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 msgr2=0x7f41880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:41.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 --2- 172.21.15.137:0/2247029107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41880f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4178009fb0 tx=0x7f4178009b30).stop 2023-11-15T06:01:41.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 -- 172.21.15.137:0/2247029107 shutdown_connections 2023-11-15T06:01:41.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 --2- 172.21.15.137:0/2247029107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41880f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:41.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 -- 172.21.15.137:0/2247029107 >> 172.21.15.137:0/2247029107 conn(0x7f41880f09a0 msgr2=0x7f41880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:41.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 -- 172.21.15.137:0/2247029107 shutdown_connections 2023-11-15T06:01:41.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 -- 172.21.15.137:0/2247029107 wait complete. 2023-11-15T06:01:41.200 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.197+0000 7f418ca17700 1 Processor -- start 2023-11-15T06:01:41.201 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f418ca17700 1 -- start start 2023-11-15T06:01:41.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f418ca17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:41.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f418ca17700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188060220 con 0x7f41880f51a0 2023-11-15T06:01:41.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f41877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:41.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f41877fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33434/0 (socket says 172.21.15.137:33434) 2023-11-15T06:01:41.203 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f41877fe700 1 -- 172.21.15.137:0/1407866921 learned_addr learned my addr 172.21.15.137:0/1407866921 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f41877fe700 1 -- 172.21.15.137:0/1407866921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178005040 con 0x7f41880f51a0 2023-11-15T06:01:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.198+0000 7f41877fe700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4178013510 tx=0x7f417800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.199+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4178009320 con 0x7f41880f51a0 2023-11-15T06:01:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.199+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4188060420 con 0x7f41880f51a0 2023-11-15T06:01:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.199+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4178013f00 con 0x7f41880f51a0 2023-11-15T06:01:41.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.199+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4178014dc0 con 0x7f41880f51a0 2023-11-15T06:01:41.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.199+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4188109880 con 0x7f41880f51a0 2023-11-15T06:01:41.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.200+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4188059100 con 0x7f41880f51a0 2023-11-15T06:01:41.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.200+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4178014420 con 0x7f41880f51a0 2023-11-15T06:01:41.205 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.200+0000 7f416ffff700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 0x7f4170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:41.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.200+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4178041020 con 0x7f41880f51a0 2023-11-15T06:01:41.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.200+0000 7f4186ffd700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 0x7f4170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:41.206 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.201+0000 7f4186ffd700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 0x7f4170032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f417c0096a0 tx=0x7f417c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:41.207 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.205+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41780095f0 con 0x7f41880f51a0 2023-11-15T06:01:41.412 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.411+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4188045ad0 con 0x7f41880f51a0 2023-11-15T06:01:41.413 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.411+0000 7f416ffff700 1 -- 172.21.15.137:0/1407866921 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f417801cb10 con 0x7f41880f51a0 2023-11-15T06:01:41.413 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:41.414 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:41.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 msgr2=0x7f4170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:41.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 0x7f4170032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f417c0096a0 tx=0x7f417c006b40).stop 2023-11-15T06:01:41.415 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 msgr2=0x7f41881091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4178013510 tx=0x7f417800b040).stop 2023-11-15T06:01:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 shutdown_connections 2023-11-15T06:01:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f417002ff60 0x7f4170032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 --2- 172.21.15.137:0/1407866921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41880f51a0 0x7f41881091d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:41.416 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 >> 172.21.15.137:0/1407866921 conn(0x7f41880f09a0 msgr2=0x7f41880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 shutdown_connections 2023-11-15T06:01:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:41.413+0000 7f418ca17700 1 -- 172.21.15.137:0/1407866921 wait complete. 2023-11-15T06:01:41.417 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:41.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:41.031358+0000 mgr.smithi003.cldzke (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:41.412980+0000 mon.smithi003 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.137:0/1407866921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:42.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:42.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:43.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3283563293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 msgr2=0x7f0a180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:43.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3283563293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a180f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f0a0c009fb0 tx=0x7f0a0c009b30).stop 2023-11-15T06:01:43.986 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3283563293 shutdown_connections 2023-11-15T06:01:43.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3283563293 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a180f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:43.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3283563293 >> 172.21.15.137:0/3283563293 conn(0x7f0a180f1c70 msgr2=0x7f0a180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:43.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3283563293 shutdown_connections 2023-11-15T06:01:43.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3283563293 wait complete. 2023-11-15T06:01:43.987 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.983+0000 7f0a1e68e700 1 Processor -- start 2023-11-15T06:01:43.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a1e68e700 1 -- start start 2023-11-15T06:01:43.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a1e68e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:43.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a1e68e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18069e20 con 0x7f0a180f6510 2023-11-15T06:01:43.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:43.988 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33446/0 (socket says 172.21.15.137:33446) 2023-11-15T06:01:43.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a17fff700 1 -- 172.21.15.137:0/3100619900 learned_addr learned my addr 172.21.15.137:0/3100619900 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:43.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a17fff700 1 -- 172.21.15.137:0/3100619900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c005040 con 0x7f0a180f6510 2023-11-15T06:01:43.989 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.984+0000 7f0a17fff700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f0a0c013510 tx=0x7f0a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:43.990 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0c009320 con 0x7f0a180f6510 2023-11-15T06:01:43.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a18069fc0 con 0x7f0a180f6510 2023-11-15T06:01:43.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a0c013f00 con 0x7f0a180f6510 2023-11-15T06:01:43.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0c014d90 con 0x7f0a180f6510 2023-11-15T06:01:43.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1806a450 con 0x7f0a180f6510 2023-11-15T06:01:43.991 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.985+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a0c0263a0 con 0x7f0a180f6510 2023-11-15T06:01:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.986+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a18059100 con 0x7f0a180f6510 2023-11-15T06:01:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.986+0000 7f0a14ff9700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 0x7f0a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.986+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a0c0413a0 con 0x7f0a180f6510 2023-11-15T06:01:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.986+0000 7f0a177fe700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 0x7f0a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.986+0000 7f0a177fe700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 0x7f0a00032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0a080096a0 tx=0x7f0a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:43.993 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:43.990+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a0c0126f0 con 0x7f0a180f6510 2023-11-15T06:01:44.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.193+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a180fbf60 con 0x7f0a180f6510 2023-11-15T06:01:44.195 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.194+0000 7f0a14ff9700 1 -- 172.21.15.137:0/3100619900 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0a0c012330 con 0x7f0a180f6510 2023-11-15T06:01:44.196 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:44.196 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:44.197 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 msgr2=0x7f0a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:44.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 0x7f0a00032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0a080096a0 tx=0x7f0a08006b40).stop 2023-11-15T06:01:44.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 msgr2=0x7f0a1806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:44.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f0a0c013510 tx=0x7f0a0c00b040).stop 2023-11-15T06:01:44.198 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 shutdown_connections 2023-11-15T06:01:44.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0a0002ff60 0x7f0a00032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:44.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 --2- 172.21.15.137:0/3100619900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0a180f6510 0x7f0a1806b9a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:44.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 >> 172.21.15.137:0/3100619900 conn(0x7f0a180f1c70 msgr2=0x7f0a180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:44.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 shutdown_connections 2023-11-15T06:01:44.199 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:44.196+0000 7f0a1e68e700 1 -- 172.21.15.137:0/3100619900 wait complete. 2023-11-15T06:01:44.200 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:44.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:44 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:43.031682+0000 mgr.smithi003.cldzke (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:45.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:45 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:44.195496+0000 mon.smithi003 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.137:0/3100619900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:45.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:45.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:45.032002+0000 mgr.smithi003.cldzke (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:46.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- 172.21.15.137:0/975674761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 msgr2=0x7f5b880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:46.724 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 --2- 172.21.15.137:0/975674761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b880f66c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f5b84009fb0 tx=0x7f5b84009b30).stop 2023-11-15T06:01:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- 172.21.15.137:0/975674761 shutdown_connections 2023-11-15T06:01:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 --2- 172.21.15.137:0/975674761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b880f66c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- 172.21.15.137:0/975674761 >> 172.21.15.137:0/975674761 conn(0x7f5b880f1aa0 msgr2=0x7f5b880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- 172.21.15.137:0/975674761 shutdown_connections 2023-11-15T06:01:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- 172.21.15.137:0/975674761 wait complete. 2023-11-15T06:01:46.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 Processor -- start 2023-11-15T06:01:46.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.722+0000 7f5b8e783700 1 -- start start 2023-11-15T06:01:46.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8e783700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:46.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8e783700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881069a0 con 0x7f5b880f62a0 2023-11-15T06:01:46.726 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8d781700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:46.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8d781700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:33454/0 (socket says 172.21.15.137:33454) 2023-11-15T06:01:46.727 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8d781700 1 -- 172.21.15.137:0/1881520104 learned_addr learned my addr 172.21.15.137:0/1881520104 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:46.728 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8d781700 1 -- 172.21.15.137:0/1881520104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84005040 con 0x7f5b880f62a0 2023-11-15T06:01:46.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b8d781700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5b84013510 tx=0x7f5b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:46.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.723+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b84009320 con 0x7f5b880f62a0 2023-11-15T06:01:46.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.724+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b88106ba0 con 0x7f5b880f62a0 2023-11-15T06:01:46.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.724+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b84013f00 con 0x7f5b880f62a0 2023-11-15T06:01:46.729 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.724+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b84014dc0 con 0x7f5b880f62a0 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.724+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b88107030 con 0x7f5b880f62a0 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b88045ad0 con 0x7f5b880f62a0 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b840095f0 con 0x7f5b880f62a0 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b7dffb700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b840410c0 con 0x7f5b880f62a0 2023-11-15T06:01:46.730 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b8cf80700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:46.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.725+0000 7f5b8cf80700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 0x7f5b74032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5b780096a0 tx=0x7f5b78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:46.731 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.730+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b84014420 con 0x7f5b880f62a0 2023-11-15T06:01:46.938 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.936+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b88107ee0 con 0x7f5b880f62a0 2023-11-15T06:01:46.939 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.937+0000 7f5b7dffb700 1 -- 172.21.15.137:0/1881520104 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5b84010230 con 0x7f5b880f62a0 2023-11-15T06:01:46.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:46.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:46.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 msgr2=0x7f5b74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:46.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 0x7f5b74032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5b780096a0 tx=0x7f5b78006b40).stop 2023-11-15T06:01:46.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 msgr2=0x7f5b881064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:46.941 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5b84013510 tx=0x7f5b8400b040).stop 2023-11-15T06:01:46.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 shutdown_connections 2023-11-15T06:01:46.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f5b7402ff60 0x7f5b74032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:46.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 --2- 172.21.15.137:0/1881520104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b880f62a0 0x7f5b881064b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:46.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 >> 172.21.15.137:0/1881520104 conn(0x7f5b880f1aa0 msgr2=0x7f5b880f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:46.942 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 shutdown_connections 2023-11-15T06:01:46.943 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:46.939+0000 7f5b8e783700 1 -- 172.21.15.137:0/1881520104 wait complete. 2023-11-15T06:01:46.943 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:47.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:47 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:46.938763+0000 mon.smithi003 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.137:0/1881520104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:48.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:48.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:48.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:47.032256+0000 mgr.smithi003.cldzke (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:49.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- 172.21.15.137:0/3994646633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 msgr2=0x7f1ca00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:49.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 --2- 172.21.15.137:0/3994646633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca00f90f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1c90009fb0 tx=0x7f1c90009b30).stop 2023-11-15T06:01:49.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- 172.21.15.137:0/3994646633 shutdown_connections 2023-11-15T06:01:49.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 --2- 172.21.15.137:0/3994646633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca00f90f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:49.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- 172.21.15.137:0/3994646633 >> 172.21.15.137:0/3994646633 conn(0x7f1ca00f0b70 msgr2=0x7f1ca00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- 172.21.15.137:0/3994646633 shutdown_connections 2023-11-15T06:01:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- 172.21.15.137:0/3994646633 wait complete. 2023-11-15T06:01:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 Processor -- start 2023-11-15T06:01:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- start start 2023-11-15T06:01:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:49.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.492+0000 7f1ca6971700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca0106bd0 con 0x7f1ca0060280 2023-11-15T06:01:49.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:49.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50918/0 (socket says 172.21.15.137:50918) 2023-11-15T06:01:49.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9ffff700 1 -- 172.21.15.137:0/2492902887 learned_addr learned my addr 172.21.15.137:0/2492902887 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:49.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9ffff700 1 -- 172.21.15.137:0/2492902887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90005040 con 0x7f1ca0060280 2023-11-15T06:01:49.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9ffff700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1c90013510 tx=0x7f1c9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:49.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c90009320 con 0x7f1ca0060280 2023-11-15T06:01:49.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca0106dd0 con 0x7f1ca0060280 2023-11-15T06:01:49.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c90013f00 con 0x7f1ca0060280 2023-11-15T06:01:49.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca0107260 con 0x7f1ca0060280 2023-11-15T06:01:49.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.493+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c90014dc0 con 0x7f1ca0060280 2023-11-15T06:01:49.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.494+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c900095f0 con 0x7f1ca0060280 2023-11-15T06:01:49.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.494+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca0045ad0 con 0x7f1ca0060280 2023-11-15T06:01:49.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.494+0000 7f1c9cff9700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:49.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.494+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c900410e0 con 0x7f1ca0060280 2023-11-15T06:01:49.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.495+0000 7f1c9f7fe700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:49.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.495+0000 7f1c9f7fe700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 0x7f1c88032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1c940096a0 tx=0x7f1c94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:49.501 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.499+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c90014420 con 0x7f1ca0060280 2023-11-15T06:01:49.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.706+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ca01080b0 con 0x7f1ca0060280 2023-11-15T06:01:49.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.706+0000 7f1c9cff9700 1 -- 172.21.15.137:0/2492902887 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1c9001c920 con 0x7f1ca0060280 2023-11-15T06:01:49.708 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:49.708 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:49.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 msgr2=0x7f1c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:49.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 0x7f1c88032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1c940096a0 tx=0x7f1c94006b40).stop 2023-11-15T06:01:49.710 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 msgr2=0x7f1ca01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:49.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1c90013510 tx=0x7f1c9000b040).stop 2023-11-15T06:01:49.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 shutdown_connections 2023-11-15T06:01:49.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c8802ff60 0x7f1c88032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:49.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 --2- 172.21.15.137:0/2492902887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ca0060280 0x7f1ca01066e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:49.711 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 >> 172.21.15.137:0/2492902887 conn(0x7f1ca00f0b70 msgr2=0x7f1ca00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:49.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 shutdown_connections 2023-11-15T06:01:49.712 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:49.708+0000 7f1ca6971700 1 -- 172.21.15.137:0/2492902887 wait complete. 2023-11-15T06:01:49.712 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:50.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:49.032648+0000 mgr.smithi003.cldzke (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:49.707888+0000 mon.smithi003 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.137:0/2492902887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:51.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:51.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:51.033053+0000 mgr.smithi003.cldzke (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:52.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 -- 172.21.15.137:0/3361711938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 msgr2=0x7fe90c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:52.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 --2- 172.21.15.137:0/3361711938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fe900009fb0 tx=0x7fe900009b30).stop 2023-11-15T06:01:52.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 -- 172.21.15.137:0/3361711938 shutdown_connections 2023-11-15T06:01:52.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 --2- 172.21.15.137:0/3361711938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:52.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 -- 172.21.15.137:0/3361711938 >> 172.21.15.137:0/3361711938 conn(0x7fe90c0f1c70 msgr2=0x7fe90c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:52.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 -- 172.21.15.137:0/3361711938 shutdown_connections 2023-11-15T06:01:52.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.257+0000 7fe912f37700 1 -- 172.21.15.137:0/3361711938 wait complete. 2023-11-15T06:01:52.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe912f37700 1 Processor -- start 2023-11-15T06:01:52.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe912f37700 1 -- start start 2023-11-15T06:01:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe912f37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe912f37700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c069e20 con 0x7fe90c0f6510 2023-11-15T06:01:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe910cd3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe910cd3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50932/0 (socket says 172.21.15.137:50932) 2023-11-15T06:01:52.263 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe910cd3700 1 -- 172.21.15.137:0/1610918355 learned_addr learned my addr 172.21.15.137:0/1610918355 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.258+0000 7fe910cd3700 1 -- 172.21.15.137:0/1610918355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900005040 con 0x7fe90c0f6510 2023-11-15T06:01:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe910cd3700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe900014000 tx=0x7fe90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9000095f0 con 0x7fe90c0f6510 2023-11-15T06:01:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c069fc0 con 0x7fe90c0f6510 2023-11-15T06:01:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe900007e70 con 0x7fe90c0f6510 2023-11-15T06:01:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe900014ea0 con 0x7fe90c0f6510 2023-11-15T06:01:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.259+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c06a450 con 0x7fe90c0f6510 2023-11-15T06:01:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe900025070 con 0x7fe90c0f6510 2023-11-15T06:01:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c059100 con 0x7fe90c0f6510 2023-11-15T06:01:52.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe9097fa700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:52.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe900040a30 con 0x7fe90c0f6510 2023-11-15T06:01:52.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe90bfff700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:52.267 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.260+0000 7fe90bfff700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 0x7fe8fc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:52.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.265+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9000079c0 con 0x7fe90c0f6510 2023-11-15T06:01:52.475 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.473+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe90c0fbf60 con 0x7fe90c0f6510 2023-11-15T06:01:52.476 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe9097fa700 1 -- 172.21.15.137:0/1610918355 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe90000f300 con 0x7fe90c0f6510 2023-11-15T06:01:52.477 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:52.477 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:52.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 msgr2=0x7fe8fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:52.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 0x7fe8fc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).stop 2023-11-15T06:01:52.479 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 msgr2=0x7fe90c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:52.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fe900014000 tx=0x7fe90000b040).stop 2023-11-15T06:01:52.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 shutdown_connections 2023-11-15T06:01:52.480 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe8fc02ff60 0x7fe8fc032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:52.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 --2- 172.21.15.137:0/1610918355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe90c0f6510 0x7fe90c06b9a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:52.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 >> 172.21.15.137:0/1610918355 conn(0x7fe90c0f1c70 msgr2=0x7fe90c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:52.481 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 shutdown_connections 2023-11-15T06:01:52.482 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:52.475+0000 7fe912f37700 1 -- 172.21.15.137:0/1610918355 wait complete. 2023-11-15T06:01:52.482 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:52.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:52 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:52.475362+0000 mon.smithi003 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.137:0/1610918355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:53.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:53.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:53.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:53 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:53.033453+0000 mgr.smithi003.cldzke (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:55.109 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 -- 172.21.15.137:0/821317923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 msgr2=0x7f78680f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:55.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 --2- 172.21.15.137:0/821317923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78680f4b60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f785c009fb0 tx=0x7f785c009b30).stop 2023-11-15T06:01:55.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 -- 172.21.15.137:0/821317923 shutdown_connections 2023-11-15T06:01:55.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 --2- 172.21.15.137:0/821317923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78680f4b60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:55.110 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 -- 172.21.15.137:0/821317923 >> 172.21.15.137:0/821317923 conn(0x7f78680f0260 msgr2=0x7f78680f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:55.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 -- 172.21.15.137:0/821317923 shutdown_connections 2023-11-15T06:01:55.111 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.107+0000 7f786ca29700 1 -- 172.21.15.137:0/821317923 wait complete. 2023-11-15T06:01:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.108+0000 7f786ca29700 1 Processor -- start 2023-11-15T06:01:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.108+0000 7f786ca29700 1 -- start start 2023-11-15T06:01:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.108+0000 7f786ca29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.108+0000 7f786ca29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7868106bc0 con 0x7f78680f4740 2023-11-15T06:01:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:55.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50940/0 (socket says 172.21.15.137:50940) 2023-11-15T06:01:55.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786659c700 1 -- 172.21.15.137:0/3218843187 learned_addr learned my addr 172.21.15.137:0/3218843187 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:55.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786659c700 1 -- 172.21.15.137:0/3218843187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c005040 con 0x7f78680f4740 2023-11-15T06:01:55.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786659c700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f785c013510 tx=0x7f785c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:55.114 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785c009320 con 0x7f78680f4740 2023-11-15T06:01:55.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f785c013f00 con 0x7f78680f4740 2023-11-15T06:01:55.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7868106dc0 con 0x7f78680f4740 2023-11-15T06:01:55.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785c014d90 con 0x7f78680f4740 2023-11-15T06:01:55.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.109+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7868107250 con 0x7f78680f4740 2023-11-15T06:01:55.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.110+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f785c0095f0 con 0x7f78680f4740 2023-11-15T06:01:55.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.110+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7868045ad0 con 0x7f78680f4740 2023-11-15T06:01:55.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.110+0000 7f7856ffd700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 0x7f7850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:55.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.110+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f785c041000 con 0x7f78680f4740 2023-11-15T06:01:55.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.111+0000 7f7865d9b700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 0x7f7850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:55.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.111+0000 7f7865d9b700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 0x7f7850032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f78580096a0 tx=0x7f7858006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:55.117 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.116+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f785c014420 con 0x7f78680f4740 2023-11-15T06:01:55.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.318+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7868108100 con 0x7f78680f4740 2023-11-15T06:01:55.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.319+0000 7f7856ffd700 1 -- 172.21.15.137:0/3218843187 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f785c01d7f0 con 0x7f78680f4740 2023-11-15T06:01:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:55.321 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:55.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 msgr2=0x7f7850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:55.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 0x7f7850032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f78580096a0 tx=0x7f7858006b40).stop 2023-11-15T06:01:55.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 msgr2=0x7f78681066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:55.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f785c013510 tx=0x7f785c00b040).stop 2023-11-15T06:01:55.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 shutdown_connections 2023-11-15T06:01:55.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f785002ff60 0x7f7850032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:55.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 --2- 172.21.15.137:0/3218843187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f78680f4740 0x7f78681066d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:55.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 >> 172.21.15.137:0/3218843187 conn(0x7f78680f0260 msgr2=0x7f78680f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:55.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 shutdown_connections 2023-11-15T06:01:55.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:55.321+0000 7f786ca29700 1 -- 172.21.15.137:0/3218843187 wait complete. 2023-11-15T06:01:55.325 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:55.033775+0000 mgr.smithi003.cldzke (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:55.320550+0000 mon.smithi003 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.137:0/3218843187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:56.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:56.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:01:57.847 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 -- 172.21.15.137:0/2245456725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 msgr2=0x7f9b100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:57.848 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 --2- 172.21.15.137:0/2245456725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b100f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f9b04009fb0 tx=0x7f9b04009b30).stop 2023-11-15T06:01:57.848 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 -- 172.21.15.137:0/2245456725 shutdown_connections 2023-11-15T06:01:57.848 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 --2- 172.21.15.137:0/2245456725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b100f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:57.848 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 -- 172.21.15.137:0/2245456725 >> 172.21.15.137:0/2245456725 conn(0x7f9b100f1c70 msgr2=0x7f9b100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:57.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 -- 172.21.15.137:0/2245456725 shutdown_connections 2023-11-15T06:01:57.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.846+0000 7f9b1708d700 1 -- 172.21.15.137:0/2245456725 wait complete. 2023-11-15T06:01:57.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.847+0000 7f9b1708d700 1 Processor -- start 2023-11-15T06:01:57.849 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.847+0000 7f9b1708d700 1 -- start start 2023-11-15T06:01:57.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.847+0000 7f9b1708d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:57.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.847+0000 7f9b1708d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10106c20 con 0x7f9b100f6510 2023-11-15T06:01:57.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b14e29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:57.851 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b14e29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:50960/0 (socket says 172.21.15.137:50960) 2023-11-15T06:01:57.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b14e29700 1 -- 172.21.15.137:0/4035963314 learned_addr learned my addr 172.21.15.137:0/4035963314 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:01:57.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b14e29700 1 -- 172.21.15.137:0/4035963314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04005040 con 0x7f9b100f6510 2023-11-15T06:01:57.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b14e29700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9b04013510 tx=0x7f9b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:57.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b04009320 con 0x7f9b100f6510 2023-11-15T06:01:57.852 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b10106e20 con 0x7f9b100f6510 2023-11-15T06:01:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b04013f00 con 0x7f9b100f6510 2023-11-15T06:01:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b04014d90 con 0x7f9b100f6510 2023-11-15T06:01:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.848+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b101072b0 con 0x7f9b100f6510 2023-11-15T06:01:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.849+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b04014420 con 0x7f9b100f6510 2023-11-15T06:01:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.849+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b10045ad0 con 0x7f9b100f6510 2023-11-15T06:01:57.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.849+0000 7f9b0d7fa700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 0x7f9b00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:01:57.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.849+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b040410c0 con 0x7f9b100f6510 2023-11-15T06:01:57.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.850+0000 7f9b0ffff700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 0x7f9b00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:01:57.854 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.850+0000 7f9b0ffff700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 0x7f9b00032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9afc0096a0 tx=0x7f9afc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:01:57.855 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:57.854+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b040095f0 con 0x7f9b100f6510 2023-11-15T06:01:58.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.059+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b100fbf60 con 0x7f9b100f6510 2023-11-15T06:01:58.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.060+0000 7f9b0d7fa700 1 -- 172.21.15.137:0/4035963314 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9b0401db70 con 0x7f9b100f6510 2023-11-15T06:01:58.061 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:01:58.061 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:01:58.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 msgr2=0x7f9b00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:58.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 0x7f9b00032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9afc0096a0 tx=0x7f9afc006b40).stop 2023-11-15T06:01:58.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 msgr2=0x7f9b10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:01:58.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9b04013510 tx=0x7f9b0400b040).stop 2023-11-15T06:01:58.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 shutdown_connections 2023-11-15T06:01:58.063 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9b0002ff60 0x7f9b00032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:58.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 --2- 172.21.15.137:0/4035963314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b100f6510 0x7f9b10106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:01:58.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.061+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 >> 172.21.15.137:0/4035963314 conn(0x7f9b100f1c70 msgr2=0x7f9b100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:01:58.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.062+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 shutdown_connections 2023-11-15T06:01:58.064 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:01:58.062+0000 7f9b1708d700 1 -- 172.21.15.137:0/4035963314 wait complete. 2023-11-15T06:01:58.065 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:01:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:57.034118+0000 mgr.smithi003.cldzke (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:01:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:01:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:01:58.060227+0000 mon.smithi003 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.137:0/4035963314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:01:59.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:01:59.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:00.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:01:59.034476+0000 mgr.smithi003.cldzke (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:00.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 -- 172.21.15.137:0/522760489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 msgr2=0x7f9adc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:00.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 --2- 172.21.15.137:0/522760489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc0f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f9ac4009fb0 tx=0x7f9ac4009b30).stop 2023-11-15T06:02:00.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 -- 172.21.15.137:0/522760489 shutdown_connections 2023-11-15T06:02:00.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 --2- 172.21.15.137:0/522760489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc0f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:00.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 -- 172.21.15.137:0/522760489 >> 172.21.15.137:0/522760489 conn(0x7f9adc0f1c70 msgr2=0x7f9adc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:00.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 -- 172.21.15.137:0/522760489 shutdown_connections 2023-11-15T06:02:00.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.545+0000 7f9ae136a700 1 -- 172.21.15.137:0/522760489 wait complete. 2023-11-15T06:02:00.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9ae136a700 1 Processor -- start 2023-11-15T06:02:00.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9ae136a700 1 -- start start 2023-11-15T06:02:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9ae136a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9ae136a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc069e20 con 0x7f9adc0f6510 2023-11-15T06:02:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9adaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9adaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54708/0 (socket says 172.21.15.137:54708) 2023-11-15T06:02:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.546+0000 7f9adaffd700 1 -- 172.21.15.137:0/2503160429 learned_addr learned my addr 172.21.15.137:0/2503160429 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:00.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9adaffd700 1 -- 172.21.15.137:0/2503160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4005040 con 0x7f9adc0f6510 2023-11-15T06:02:00.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9adaffd700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f9ac4013510 tx=0x7f9ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:00.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac4009320 con 0x7f9adc0f6510 2023-11-15T06:02:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc069fc0 con 0x7f9adc0f6510 2023-11-15T06:02:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ac4013f00 con 0x7f9adc0f6510 2023-11-15T06:02:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac4014d90 con 0x7f9adc0f6510 2023-11-15T06:02:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.547+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc06a450 con 0x7f9adc0f6510 2023-11-15T06:02:00.552 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.548+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ac4014420 con 0x7f9adc0f6510 2023-11-15T06:02:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.548+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc059100 con 0x7f9adc0f6510 2023-11-15T06:02:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.548+0000 7f9ad37fe700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 0x7f9ac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.548+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ac4041110 con 0x7f9adc0f6510 2023-11-15T06:02:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.548+0000 7f9ada7fc700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 0x7f9ac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:00.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.549+0000 7f9ada7fc700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 0x7f9ac8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:00.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.553+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ac40095f0 con 0x7f9adc0f6510 2023-11-15T06:02:00.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.756+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9adc0fbf60 con 0x7f9adc0f6510 2023-11-15T06:02:00.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.757+0000 7f9ad37fe700 1 -- 172.21.15.137:0/2503160429 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9ac401db70 con 0x7f9adc0f6510 2023-11-15T06:02:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:00.759 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:00.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 msgr2=0x7f9ac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:00.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 0x7f9ac8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).stop 2023-11-15T06:02:00.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 msgr2=0x7f9adc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:00.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f9ac4013510 tx=0x7f9ac400b040).stop 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 shutdown_connections 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f9ac802ff60 0x7f9ac8032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 --2- 172.21.15.137:0/2503160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9adc0f6510 0x7f9adc06b9a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 >> 172.21.15.137:0/2503160429 conn(0x7f9adc0f1c70 msgr2=0x7f9adc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 shutdown_connections 2023-11-15T06:02:00.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:00.759+0000 7f9ae136a700 1 -- 172.21.15.137:0/2503160429 wait complete. 2023-11-15T06:02:00.762 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:01.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:00.757595+0000 mon.smithi003 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.137:0/2503160429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:01.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:01.034895+0000 mgr.smithi003.cldzke (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:02.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:02.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:03.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.314+0000 7fb1795fd700 1 -- 172.21.15.137:0/1368682254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 msgr2=0x7fb1740f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:03.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.314+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1368682254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb1740f4be0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fb168009fb0 tx=0x7fb168009b30).stop 2023-11-15T06:02:03.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.314+0000 7fb1795fd700 1 -- 172.21.15.137:0/1368682254 shutdown_connections 2023-11-15T06:02:03.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.314+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1368682254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb1740f4be0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:03.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.314+0000 7fb1795fd700 1 -- 172.21.15.137:0/1368682254 >> 172.21.15.137:0/1368682254 conn(0x7fb1740f02e0 msgr2=0x7fb1740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:03.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 -- 172.21.15.137:0/1368682254 shutdown_connections 2023-11-15T06:02:03.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 -- 172.21.15.137:0/1368682254 wait complete. 2023-11-15T06:02:03.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 Processor -- start 2023-11-15T06:02:03.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 -- start start 2023-11-15T06:02:03.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:03.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.315+0000 7fb1795fd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174106bf0 con 0x7fb1740f47c0 2023-11-15T06:02:03.323 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.316+0000 7fb172ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.316+0000 7fb172ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54732/0 (socket says 172.21.15.137:54732) 2023-11-15T06:02:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.316+0000 7fb172ffd700 1 -- 172.21.15.137:0/1489119101 learned_addr learned my addr 172.21.15.137:0/1489119101 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.316+0000 7fb172ffd700 1 -- 172.21.15.137:0/1489119101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168005040 con 0x7fb1740f47c0 2023-11-15T06:02:03.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.316+0000 7fb172ffd700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb168013510 tx=0x7fb16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:03.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb168009320 con 0x7fb1740f47c0 2023-11-15T06:02:03.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb174106df0 con 0x7fb1740f47c0 2023-11-15T06:02:03.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb168013f00 con 0x7fb1740f47c0 2023-11-15T06:02:03.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb168014d90 con 0x7fb1740f47c0 2023-11-15T06:02:03.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb174107280 con 0x7fb1740f47c0 2023-11-15T06:02:03.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.317+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1680263a0 con 0x7fb1740f47c0 2023-11-15T06:02:03.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.318+0000 7fb15b7fe700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 0x7fb15c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:03.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.318+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb174045ad0 con 0x7fb1740f47c0 2023-11-15T06:02:03.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.318+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1680413d0 con 0x7fb1740f47c0 2023-11-15T06:02:03.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.318+0000 7fb1727fc700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 0x7fb15c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:03.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.318+0000 7fb1727fc700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 0x7fb15c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb1640096a0 tx=0x7fb164006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:03.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.323+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb168012920 con 0x7fb1740f47c0 2023-11-15T06:02:03.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.533+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb174108130 con 0x7fb1740f47c0 2023-11-15T06:02:03.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.533+0000 7fb15b7fe700 1 -- 172.21.15.137:0/1489119101 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb16802b030 con 0x7fb1740f47c0 2023-11-15T06:02:03.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:03.535 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:03.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 msgr2=0x7fb15c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:03.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 0x7fb15c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fb1640096a0 tx=0x7fb164006b40).stop 2023-11-15T06:02:03.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 msgr2=0x7fb174106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:03.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fb168013510 tx=0x7fb16800b040).stop 2023-11-15T06:02:03.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 shutdown_connections 2023-11-15T06:02:03.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb15c02ff60 0x7fb15c032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:03.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 --2- 172.21.15.137:0/1489119101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1740f47c0 0x7fb174106700 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:03.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 >> 172.21.15.137:0/1489119101 conn(0x7fb1740f02e0 msgr2=0x7fb1740f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:03.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 shutdown_connections 2023-11-15T06:02:03.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:03.535+0000 7fb1795fd700 1 -- 172.21.15.137:0/1489119101 wait complete. 2023-11-15T06:02:03.538 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:04.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:03.035213+0000 mgr.smithi003.cldzke (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:04.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:03.533860+0000 mon.smithi003 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.137:0/1489119101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:04.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:04.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:05.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T06:02:03.973197+0000 mgr.smithi003.cldzke (mgr.14164) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi003 ... 2023-11-15T06:02:05.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Verifying port 9100 ... 2023-11-15T06:02:05.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 1 from systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-15T06:02:05.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: systemctl: stderr Job for ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service failed because a timeout was exceeded. 2023-11-15T06:02:05.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: systemctl: stderr See "systemctl status ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service" and "journalctl -xe" for details. 2023-11-15T06:02:05.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T06:02:05.433 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:02:05.433 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: main() 2023-11-15T06:02:05.433 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:02:05.433 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: r = ctx.func(ctx) 2023-11-15T06:02:05.433 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:02:05.434 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: return func(ctx) 2023-11-15T06:02:05.434 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:02:05.434 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ports=daemon_ports) 2023-11-15T06:02:05.434 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:02:05.434 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: RuntimeError: Failed command: systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-15T06:02:05.435 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T06:02:05.436 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T06:02:05.436 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: yield (conn, connr) 2023-11-15T06:02:05.436 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T06:02:05.436 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: code, '\n'.join(err))) 2023-11-15T06:02:05.436 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi003 ... 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Verifying port 9100 ... 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Non-zero exit code 1 from systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: systemctl: stderr Job for ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service failed because a timeout was exceeded. 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: systemctl: stderr See "systemctl status ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service" and "journalctl -xe" for details. 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: Traceback (most recent call last): 2023-11-15T06:02:05.437 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:02:05.438 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: main() 2023-11-15T06:02:05.438 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:02:05.438 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: r = ctx.func(ctx) 2023-11-15T06:02:05.438 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: return func(ctx) 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: ports=daemon_ports) 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:02:05.439 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:02:05.440 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:02:05.440 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: File "/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:02:05.440 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:02:05.440 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: RuntimeError: Failed command: systemctl start ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-15T06:02:05.440 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cephadm 2023-11-15T06:02:03.978971+0000 mgr.smithi003.cldzke (mgr.14164) 232 : cephadm [INF] Deploying daemon prometheus.smithi003 on smithi003 2023-11-15T06:02:06.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 -- 172.21.15.137:0/3209328070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82480f5410 msgr2=0x7f82480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:06.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 --2- 172.21.15.137:0/3209328070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82480f5410 0x7f82480f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f8238009fb0 tx=0x7f8238009b30).stop 2023-11-15T06:02:06.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 -- 172.21.15.137:0/3209328070 shutdown_connections 2023-11-15T06:02:06.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 --2- 172.21.15.137:0/3209328070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82480f5410 0x7f82480f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:06.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 -- 172.21.15.137:0/3209328070 >> 172.21.15.137:0/3209328070 conn(0x7f82480f0b70 msgr2=0x7f82480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:06.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 -- 172.21.15.137:0/3209328070 shutdown_connections 2023-11-15T06:02:06.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.043+0000 7f824f59f700 1 -- 172.21.15.137:0/3209328070 wait complete. 2023-11-15T06:02:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824f59f700 1 Processor -- start 2023-11-15T06:02:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824f59f700 1 -- start start 2023-11-15T06:02:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824f59f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824f59f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481092f0 con 0x7f82481089e0 2023-11-15T06:02:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824d33b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:06.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824d33b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54746/0 (socket says 172.21.15.137:54746) 2023-11-15T06:02:06.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.044+0000 7f824d33b700 1 -- 172.21.15.137:0/2303556097 learned_addr learned my addr 172.21.15.137:0/2303556097 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:06.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f824d33b700 1 -- 172.21.15.137:0/2303556097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238005040 con 0x7f82481089e0 2023-11-15T06:02:06.049 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f824d33b700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f8238009f80 tx=0x7f8238009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:06.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82380136c0 con 0x7f82481089e0 2023-11-15T06:02:06.050 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82481094f0 con 0x7f82481089e0 2023-11-15T06:02:06.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82380095f0 con 0x7f82481089e0 2023-11-15T06:02:06.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8238014420 con 0x7f82481089e0 2023-11-15T06:02:06.051 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.045+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824810a040 con 0x7f82481089e0 2023-11-15T06:02:06.052 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.046+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8238025060 con 0x7f82481089e0 2023-11-15T06:02:06.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.046+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8248059100 con 0x7f82481089e0 2023-11-15T06:02:06.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.046+0000 7f823dffb700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 0x7f8234032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:06.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.046+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8238040d80 con 0x7f82481089e0 2023-11-15T06:02:06.053 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.046+0000 7f824cb3a700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 0x7f8234032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:06.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.047+0000 7f824cb3a700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 0x7f8234032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:06.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.051+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f823801cb60 con 0x7f82481089e0 2023-11-15T06:02:06.257 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.256+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f824810a800 con 0x7f82481089e0 2023-11-15T06:02:06.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.257+0000 7f823dffb700 1 -- 172.21.15.137:0/2303556097 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8238042070 con 0x7f82481089e0 2023-11-15T06:02:06.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:06.259 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:06.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 msgr2=0x7f8234032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 0x7f8234032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f82440096a0 tx=0x7f8244006b40).stop 2023-11-15T06:02:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 msgr2=0x7f8248108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f8238009f80 tx=0x7f8238009b30).stop 2023-11-15T06:02:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 shutdown_connections 2023-11-15T06:02:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f823402ff60 0x7f8234032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:06.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 --2- 172.21.15.137:0/2303556097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82481089e0 0x7f8248108e00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:06.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 >> 172.21.15.137:0/2303556097 conn(0x7f82480f0b70 msgr2=0x7f824805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:06.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 shutdown_connections 2023-11-15T06:02:06.261 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:06.259+0000 7f824f59f700 1 -- 172.21.15.137:0/2303556097 wait complete. 2023-11-15T06:02:06.262 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:06 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:05.035593+0000 mgr.smithi003.cldzke (mgr.14164) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:07.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:07 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:06.257612+0000 mon.smithi003 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.137:0/2303556097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:07.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:07.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:07.035976+0000 mgr.smithi003.cldzke (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:08.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 -- 172.21.15.137:0/213655491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 msgr2=0x7f85280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:08.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 --2- 172.21.15.137:0/213655491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f85280f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8510009fb0 tx=0x7f8510009b30).stop 2023-11-15T06:02:08.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 -- 172.21.15.137:0/213655491 shutdown_connections 2023-11-15T06:02:08.884 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 --2- 172.21.15.137:0/213655491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f85280f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:08.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 -- 172.21.15.137:0/213655491 >> 172.21.15.137:0/213655491 conn(0x7f85280f1c70 msgr2=0x7f85280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:08.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 -- 172.21.15.137:0/213655491 shutdown_connections 2023-11-15T06:02:08.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.882+0000 7f852e1ef700 1 -- 172.21.15.137:0/213655491 wait complete. 2023-11-15T06:02:08.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.883+0000 7f852e1ef700 1 Processor -- start 2023-11-15T06:02:08.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.883+0000 7f852e1ef700 1 -- start start 2023-11-15T06:02:08.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.883+0000 7f852e1ef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:08.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.883+0000 7f852e1ef700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8528069e20 con 0x7f85280f6510 2023-11-15T06:02:08.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f85277fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:08.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f85277fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54772/0 (socket says 172.21.15.137:54772) 2023-11-15T06:02:08.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f85277fe700 1 -- 172.21.15.137:0/2776532799 learned_addr learned my addr 172.21.15.137:0/2776532799 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:08.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f85277fe700 1 -- 172.21.15.137:0/2776532799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8510005040 con 0x7f85280f6510 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f85277fe700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8510013510 tx=0x7f851000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8510009320 con 0x7f85280f6510 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8528069fc0 con 0x7f85280f6510 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8510013f00 con 0x7f85280f6510 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8510014d90 con 0x7f85280f6510 2023-11-15T06:02:08.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.884+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852806a450 con 0x7f85280f6510 2023-11-15T06:02:08.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.885+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85100095f0 con 0x7f85280f6510 2023-11-15T06:02:08.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.885+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528059100 con 0x7f85280f6510 2023-11-15T06:02:08.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.885+0000 7f851ffff700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:08.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.885+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85100410a0 con 0x7f85280f6510 2023-11-15T06:02:08.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.885+0000 7f8526ffd700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:08.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.886+0000 7f8526ffd700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f85180096a0 tx=0x7f8518006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:08.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:08.890+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8510010230 con 0x7f85280f6510 2023-11-15T06:02:09.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.093+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8528045ad0 con 0x7f85280f6510 2023-11-15T06:02:09.094 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.094+0000 7f851ffff700 1 -- 172.21.15.137:0/2776532799 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8510010230 con 0x7f85280f6510 2023-11-15T06:02:09.095 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:09.095 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:09.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 msgr2=0x7f8514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:09.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f85180096a0 tx=0x7f8518006b40).stop 2023-11-15T06:02:09.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 msgr2=0x7f852806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:09.096 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8510013510 tx=0x7f851000b040).stop 2023-11-15T06:02:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 shutdown_connections 2023-11-15T06:02:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 --2- 172.21.15.137:0/2776532799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85280f6510 0x7f852806b9a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 >> 172.21.15.137:0/2776532799 conn(0x7f85280f1c70 msgr2=0x7f85280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 shutdown_connections 2023-11-15T06:02:09.098 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:09.095+0000 7f852e1ef700 1 -- 172.21.15.137:0/2776532799 wait complete. 2023-11-15T06:02:09.098 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:09.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:09 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:09.094102+0000 mon.smithi003 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.137:0/2776532799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:10.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:09.036297+0000 mgr.smithi003.cldzke (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:10.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:10.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:11.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 -- 172.21.15.137:0/141397763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 msgr2=0x7f35300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:11.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 --2- 172.21.15.137:0/141397763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f35300f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3518009fb0 tx=0x7f3518009b30).stop 2023-11-15T06:02:11.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 -- 172.21.15.137:0/141397763 shutdown_connections 2023-11-15T06:02:11.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 --2- 172.21.15.137:0/141397763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f35300f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:11.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 -- 172.21.15.137:0/141397763 >> 172.21.15.137:0/141397763 conn(0x7f35300f0b70 msgr2=0x7f35300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:11.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 -- 172.21.15.137:0/141397763 shutdown_connections 2023-11-15T06:02:11.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.681+0000 7f353526e700 1 -- 172.21.15.137:0/141397763 wait complete. 2023-11-15T06:02:11.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f353526e700 1 Processor -- start 2023-11-15T06:02:11.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f353526e700 1 -- start start 2023-11-15T06:02:11.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f353526e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:11.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f353526e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530105af0 con 0x7f35300f5410 2023-11-15T06:02:11.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f352effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:11.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f352effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34454/0 (socket says 172.21.15.137:34454) 2023-11-15T06:02:11.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f352effd700 1 -- 172.21.15.137:0/3530026218 learned_addr learned my addr 172.21.15.137:0/3530026218 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:11.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f352effd700 1 -- 172.21.15.137:0/3530026218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518005040 con 0x7f35300f5410 2023-11-15T06:02:11.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.682+0000 7f352effd700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3518013510 tx=0x7f351800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:11.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.683+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3518009320 con 0x7f35300f5410 2023-11-15T06:02:11.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.683+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3530105cf0 con 0x7f35300f5410 2023-11-15T06:02:11.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.683+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3518013f00 con 0x7f35300f5410 2023-11-15T06:02:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.683+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3518014d90 con 0x7f35300f5410 2023-11-15T06:02:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.683+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3530106180 con 0x7f35300f5410 2023-11-15T06:02:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f351800f410 con 0x7f35300f5410 2023-11-15T06:02:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f35277fe700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 0x7f351c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3518042e60 con 0x7f35300f5410 2023-11-15T06:02:11.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3530059100 con 0x7f35300f5410 2023-11-15T06:02:11.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f352e7fc700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 0x7f351c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:11.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.684+0000 7f352e7fc700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 0x7f351c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f35200096a0 tx=0x7f3520006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:11.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.689+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3518012920 con 0x7f35300f5410 2023-11-15T06:02:11.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.892+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3530045ad0 con 0x7f35300f5410 2023-11-15T06:02:11.893 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.892+0000 7f35277fe700 1 -- 172.21.15.137:0/3530026218 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3518012330 con 0x7f35300f5410 2023-11-15T06:02:11.893 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:11.894 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:11.894 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 msgr2=0x7f351c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:11.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 0x7f351c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f35200096a0 tx=0x7f3520006b40).stop 2023-11-15T06:02:11.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 msgr2=0x7f3530105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:11.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3518013510 tx=0x7f351800b040).stop 2023-11-15T06:02:11.895 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 shutdown_connections 2023-11-15T06:02:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f351c02ffb0 0x7f351c032470 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 --2- 172.21.15.137:0/3530026218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35300f5410 0x7f3530105600 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 >> 172.21.15.137:0/3530026218 conn(0x7f35300f0b70 msgr2=0x7f35300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 shutdown_connections 2023-11-15T06:02:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:11.894+0000 7f353526e700 1 -- 172.21.15.137:0/3530026218 wait complete. 2023-11-15T06:02:11.897 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:11.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:11.036650+0000 mgr.smithi003.cldzke (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:12.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:11.892809+0000 mon.smithi003 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.137:0/3530026218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:13.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:13.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:13.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:13 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:13.037114+0000 mgr.smithi003.cldzke (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:14.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 -- 172.21.15.137:0/2944312347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 msgr2=0x7fbd140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:14.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/2944312347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd140f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fbd04009fb0 tx=0x7fbd04009b30).stop 2023-11-15T06:02:14.491 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 -- 172.21.15.137:0/2944312347 shutdown_connections 2023-11-15T06:02:14.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/2944312347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd140f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:14.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 -- 172.21.15.137:0/2944312347 >> 172.21.15.137:0/2944312347 conn(0x7fbd140f1c70 msgr2=0x7fbd140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:14.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 -- 172.21.15.137:0/2944312347 shutdown_connections 2023-11-15T06:02:14.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.488+0000 7fbd1b87c700 1 -- 172.21.15.137:0/2944312347 wait complete. 2023-11-15T06:02:14.492 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.489+0000 7fbd1b87c700 1 Processor -- start 2023-11-15T06:02:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.489+0000 7fbd1b87c700 1 -- start start 2023-11-15T06:02:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.489+0000 7fbd1b87c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.489+0000 7fbd1b87c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd14106c20 con 0x7fbd140f6510 2023-11-15T06:02:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.489+0000 7fbd19618700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd19618700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34468/0 (socket says 172.21.15.137:34468) 2023-11-15T06:02:14.494 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd19618700 1 -- 172.21.15.137:0/3238543475 learned_addr learned my addr 172.21.15.137:0/3238543475 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:14.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd19618700 1 -- 172.21.15.137:0/3238543475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04005040 con 0x7fbd140f6510 2023-11-15T06:02:14.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd19618700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fbd04014000 tx=0x7fbd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:14.495 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd040095f0 con 0x7fbd140f6510 2023-11-15T06:02:14.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd14106e20 con 0x7fbd140f6510 2023-11-15T06:02:14.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd04007e70 con 0x7fbd140f6510 2023-11-15T06:02:14.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd04014ea0 con 0x7fbd140f6510 2023-11-15T06:02:14.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.490+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd141072b0 con 0x7fbd140f6510 2023-11-15T06:02:14.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.491+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd04025070 con 0x7fbd140f6510 2023-11-15T06:02:14.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.491+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd14045ad0 con 0x7fbd140f6510 2023-11-15T06:02:14.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.491+0000 7fbd09ffb700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 0x7fbd00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:14.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.491+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd04040a40 con 0x7fbd140f6510 2023-11-15T06:02:14.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.492+0000 7fbd18e17700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 0x7fbd00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:14.497 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.492+0000 7fbd18e17700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 0x7fbd00032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbd100096a0 tx=0x7fbd10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:14.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.496+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd040079c0 con 0x7fbd140f6510 2023-11-15T06:02:14.704 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.703+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd140fbf60 con 0x7fbd140f6510 2023-11-15T06:02:14.705 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.703+0000 7fbd09ffb700 1 -- 172.21.15.137:0/3238543475 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbd0400f300 con 0x7fbd140f6510 2023-11-15T06:02:14.705 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:14.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:14.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 msgr2=0x7fbd00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:14.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 0x7fbd00032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbd100096a0 tx=0x7fbd10006b40).stop 2023-11-15T06:02:14.707 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 msgr2=0x7fbd14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fbd04014000 tx=0x7fbd0400b040).stop 2023-11-15T06:02:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 shutdown_connections 2023-11-15T06:02:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbd0002ff60 0x7fbd00032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 --2- 172.21.15.137:0/3238543475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd140f6510 0x7fbd14106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 >> 172.21.15.137:0/3238543475 conn(0x7fbd140f1c70 msgr2=0x7fbd140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 shutdown_connections 2023-11-15T06:02:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:14.705+0000 7fbd1b87c700 1 -- 172.21.15.137:0/3238543475 wait complete. 2023-11-15T06:02:14.709 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:15.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:14.704830+0000 mon.smithi003 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.137:0/3238543475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:16.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:16.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:16.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:15 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:15.037507+0000 mgr.smithi003.cldzke (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:17.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- 172.21.15.137:0/3474166271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 msgr2=0x7f80680f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:17.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 --2- 172.21.15.137:0/3474166271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f80680f8f80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f805c009fb0 tx=0x7f805c009b30).stop 2023-11-15T06:02:17.374 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- 172.21.15.137:0/3474166271 shutdown_connections 2023-11-15T06:02:17.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 --2- 172.21.15.137:0/3474166271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f80680f8f80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:17.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- 172.21.15.137:0/3474166271 >> 172.21.15.137:0/3474166271 conn(0x7f80680f09a0 msgr2=0x7f80680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:17.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- 172.21.15.137:0/3474166271 shutdown_connections 2023-11-15T06:02:17.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- 172.21.15.137:0/3474166271 wait complete. 2023-11-15T06:02:17.375 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 Processor -- start 2023-11-15T06:02:17.376 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.372+0000 7f806dbc0700 1 -- start start 2023-11-15T06:02:17.377 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806dbc0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:17.377 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806dbc0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80681069f0 con 0x7f8068060180 2023-11-15T06:02:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806cbbe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806cbbe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:34474/0 (socket says 172.21.15.137:34474) 2023-11-15T06:02:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806cbbe700 1 -- 172.21.15.137:0/4251153508 learned_addr learned my addr 172.21.15.137:0/4251153508 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806cbbe700 1 -- 172.21.15.137:0/4251153508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c005040 con 0x7f8068060180 2023-11-15T06:02:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.373+0000 7f806cbbe700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f805c013510 tx=0x7f805c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f805c009320 con 0x7f8068060180 2023-11-15T06:02:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8068106bf0 con 0x7f8068060180 2023-11-15T06:02:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f805c013f00 con 0x7f8068060180 2023-11-15T06:02:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f805c014d90 con 0x7f8068060180 2023-11-15T06:02:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8068107080 con 0x7f8068060180 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f805c0095f0 con 0x7f8068060180 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.374+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8068045ad0 con 0x7f8068060180 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.375+0000 7f80657fa700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 0x7f8058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.375+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f805c041000 con 0x7f8068060180 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.375+0000 7f8067fff700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 0x7f8058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.375+0000 7f8067fff700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 0x7f8058032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:17.381 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.380+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f805c014420 con 0x7f8068060180 2023-11-15T06:02:17.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.582+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8068107f30 con 0x7f8068060180 2023-11-15T06:02:17.584 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.582+0000 7f80657fa700 1 -- 172.21.15.137:0/4251153508 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f805c010230 con 0x7f8068060180 2023-11-15T06:02:17.585 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:17.585 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:17.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 msgr2=0x7f8058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:17.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 0x7f8058032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).stop 2023-11-15T06:02:17.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 msgr2=0x7f8068106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:17.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f805c013510 tx=0x7f805c00b040).stop 2023-11-15T06:02:17.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 shutdown_connections 2023-11-15T06:02:17.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f805802ff60 0x7f8058032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:17.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 --2- 172.21.15.137:0/4251153508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8068060180 0x7f8068106500 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:17.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 >> 172.21.15.137:0/4251153508 conn(0x7f80680f09a0 msgr2=0x7f80680f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:17.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 shutdown_connections 2023-11-15T06:02:17.588 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:17.584+0000 7f806dbc0700 1 -- 172.21.15.137:0/4251153508 wait complete. 2023-11-15T06:02:17.588 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:17.037918+0000 mgr.smithi003.cldzke (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:17.583915+0000 mon.smithi003 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.137:0/4251153508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:18.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:18.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:20.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 -- 172.21.15.137:0/2192825283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 msgr2=0x7fcbc00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:20.156 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/2192825283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f90f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fcbb4009fb0 tx=0x7fcbb4009b30).stop 2023-11-15T06:02:20.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 -- 172.21.15.137:0/2192825283 shutdown_connections 2023-11-15T06:02:20.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/2192825283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f90f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:20.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 -- 172.21.15.137:0/2192825283 >> 172.21.15.137:0/2192825283 conn(0x7fcbc00f0b70 msgr2=0x7fcbc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:20.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 -- 172.21.15.137:0/2192825283 shutdown_connections 2023-11-15T06:02:20.157 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.153+0000 7fcbc4abe700 1 -- 172.21.15.137:0/2192825283 wait complete. 2023-11-15T06:02:20.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbc4abe700 1 Processor -- start 2023-11-15T06:02:20.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbc4abe700 1 -- start start 2023-11-15T06:02:20.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbc4abe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:20.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbc4abe700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00f4ff0 con 0x7fcbc0060280 2023-11-15T06:02:20.158 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbbe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:20.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbbe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39446/0 (socket says 172.21.15.137:39446) 2023-11-15T06:02:20.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbbe59c700 1 -- 172.21.15.137:0/1187915904 learned_addr learned my addr 172.21.15.137:0/1187915904 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:20.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbbe59c700 1 -- 172.21.15.137:0/1187915904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4005040 con 0x7fcbc0060280 2023-11-15T06:02:20.159 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.154+0000 7fcbbe59c700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fcbb4013510 tx=0x7fcbb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:20.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.155+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb4009320 con 0x7fcbc0060280 2023-11-15T06:02:20.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.155+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc00f5190 con 0x7fcbc0060280 2023-11-15T06:02:20.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.155+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbb4013f00 con 0x7fcbc0060280 2023-11-15T06:02:20.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.155+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb4014dc0 con 0x7fcbc0060280 2023-11-15T06:02:20.160 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.155+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc00f5620 con 0x7fcbc0060280 2023-11-15T06:02:20.162 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0059100 con 0x7fcbc0060280 2023-11-15T06:02:20.163 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbb40095f0 con 0x7fcbc0060280 2023-11-15T06:02:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbaeffd700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 0x7fcba8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbb4041020 con 0x7fcbc0060280 2023-11-15T06:02:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbbdd9b700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 0x7fcba8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.156+0000 7fcbbdd9b700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 0x7fcba8032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcbb00096a0 tx=0x7fcbb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.160+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbb4014420 con 0x7fcbc0060280 2023-11-15T06:02:20.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.366+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbc00fbf50 con 0x7fcbc0060280 2023-11-15T06:02:20.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.366+0000 7fcbaeffd700 1 -- 172.21.15.137:0/1187915904 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcbb401c920 con 0x7fcbc0060280 2023-11-15T06:02:20.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:20.368 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:20.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 msgr2=0x7fcba8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:20.369 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 0x7fcba8032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fcbb00096a0 tx=0x7fcbb0006b40).stop 2023-11-15T06:02:20.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 msgr2=0x7fcbc00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:20.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fcbb4013510 tx=0x7fcbb400b040).stop 2023-11-15T06:02:20.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 shutdown_connections 2023-11-15T06:02:20.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fcba802ff60 0x7fcba8032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:20.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 --2- 172.21.15.137:0/1187915904 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbc0060280 0x7fcbc00f6b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:20.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 >> 172.21.15.137:0/1187915904 conn(0x7fcbc00f0b70 msgr2=0x7fcbc00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:20.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 shutdown_connections 2023-11-15T06:02:20.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:20.368+0000 7fcbc4abe700 1 -- 172.21.15.137:0/1187915904 wait complete. 2023-11-15T06:02:20.372 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:19.038300+0000 mgr.smithi003.cldzke (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:21.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:20.367788+0000 mon.smithi003 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.137:0/1187915904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:21.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:21.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:22.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:22 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:21.038683+0000 mgr.smithi003.cldzke (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:22.909 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- 172.21.15.137:0/1678152256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 msgr2=0x7efef00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:22.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 --2- 172.21.15.137:0/1678152256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef00f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7efee0009fb0 tx=0x7efee0009b30).stop 2023-11-15T06:02:22.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- 172.21.15.137:0/1678152256 shutdown_connections 2023-11-15T06:02:22.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 --2- 172.21.15.137:0/1678152256 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef00f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:22.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- 172.21.15.137:0/1678152256 >> 172.21.15.137:0/1678152256 conn(0x7efef00f0b70 msgr2=0x7efef00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:22.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- 172.21.15.137:0/1678152256 shutdown_connections 2023-11-15T06:02:22.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- 172.21.15.137:0/1678152256 wait complete. 2023-11-15T06:02:22.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 Processor -- start 2023-11-15T06:02:22.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.908+0000 7efef6630700 1 -- start start 2023-11-15T06:02:22.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efef6630700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:22.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efef6630700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0106bb0 con 0x7efef0060280 2023-11-15T06:02:22.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeeffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:22.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeeffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39466/0 (socket says 172.21.15.137:39466) 2023-11-15T06:02:22.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeeffff700 1 -- 172.21.15.137:0/615799501 learned_addr learned my addr 172.21.15.137:0/615799501 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:22.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeeffff700 1 -- 172.21.15.137:0/615799501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0005040 con 0x7efef0060280 2023-11-15T06:02:22.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeeffff700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7efee0013510 tx=0x7efee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:22.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee0009320 con 0x7efef0060280 2023-11-15T06:02:22.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef0106db0 con 0x7efef0060280 2023-11-15T06:02:22.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.909+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efee0013f00 con 0x7efef0060280 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.910+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee0014dc0 con 0x7efef0060280 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.910+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0107240 con 0x7efef0060280 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.910+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efee001d3d0 con 0x7efef0060280 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.910+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef0045ad0 con 0x7efef0060280 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.911+0000 7efeecff9700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.911+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efee0041030 con 0x7efef0060280 2023-11-15T06:02:22.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.911+0000 7efeef7fe700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:22.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.911+0000 7efeef7fe700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 0x7efed8032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:22.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:22.915+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efee00095f0 con 0x7efef0060280 2023-11-15T06:02:23.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.121+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efef01080f0 con 0x7efef0060280 2023-11-15T06:02:23.124 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.122+0000 7efeecff9700 1 -- 172.21.15.137:0/615799501 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7efee001c970 con 0x7efef0060280 2023-11-15T06:02:23.124 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:23.125 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:23.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.123+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 msgr2=0x7efed8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:23.125 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.123+0000 7efef6630700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 0x7efed8032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).stop 2023-11-15T06:02:23.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.123+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 msgr2=0x7efef01066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:23.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.123+0000 7efef6630700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7efee0013510 tx=0x7efee000b040).stop 2023-11-15T06:02:23.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 shutdown_connections 2023-11-15T06:02:23.126 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7efed802ff60 0x7efed8032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:23.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 --2- 172.21.15.137:0/615799501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efef0060280 0x7efef01066c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:23.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 >> 172.21.15.137:0/615799501 conn(0x7efef00f0b70 msgr2=0x7efef00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:23.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 shutdown_connections 2023-11-15T06:02:23.127 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:23.124+0000 7efef6630700 1 -- 172.21.15.137:0/615799501 wait complete. 2023-11-15T06:02:23.128 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:23.039018+0000 mgr.smithi003.cldzke (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:23.123581+0000 mon.smithi003 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.137:0/615799501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:24.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:24.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:25.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 -- 172.21.15.137:0/2110773515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 msgr2=0x7f0cec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:25.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 --2- 172.21.15.137:0/2110773515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec0f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f0cdc009fb0 tx=0x7f0cdc009b30).stop 2023-11-15T06:02:25.751 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 -- 172.21.15.137:0/2110773515 shutdown_connections 2023-11-15T06:02:25.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 --2- 172.21.15.137:0/2110773515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec0f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:25.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 -- 172.21.15.137:0/2110773515 >> 172.21.15.137:0/2110773515 conn(0x7f0cec0f1c70 msgr2=0x7f0cec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:25.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 -- 172.21.15.137:0/2110773515 shutdown_connections 2023-11-15T06:02:25.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.748+0000 7f0cf3279700 1 -- 172.21.15.137:0/2110773515 wait complete. 2023-11-15T06:02:25.752 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf3279700 1 Processor -- start 2023-11-15T06:02:25.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf3279700 1 -- start start 2023-11-15T06:02:25.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf3279700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:25.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf3279700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec069e20 con 0x7f0cec0f6510 2023-11-15T06:02:25.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf1015700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:25.753 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf1015700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39482/0 (socket says 172.21.15.137:39482) 2023-11-15T06:02:25.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf1015700 1 -- 172.21.15.137:0/748797208 learned_addr learned my addr 172.21.15.137:0/748797208 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:25.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.749+0000 7f0cf1015700 1 -- 172.21.15.137:0/748797208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc005040 con 0x7f0cec0f6510 2023-11-15T06:02:25.754 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0cf1015700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0cdc013510 tx=0x7f0cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:25.755 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc009320 con 0x7f0cec0f6510 2023-11-15T06:02:25.755 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec069fc0 con 0x7f0cec0f6510 2023-11-15T06:02:25.756 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cdc013f00 con 0x7f0cec0f6510 2023-11-15T06:02:25.756 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc014d90 con 0x7f0cec0f6510 2023-11-15T06:02:25.756 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.750+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec06a450 con 0x7f0cec0f6510 2023-11-15T06:02:25.756 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.751+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cdc0263a0 con 0x7f0cec0f6510 2023-11-15T06:02:25.756 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.751+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec059100 con 0x7f0cec0f6510 2023-11-15T06:02:25.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.751+0000 7f0ce1ffb700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:25.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.751+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cdc041090 con 0x7f0cec0f6510 2023-11-15T06:02:25.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.751+0000 7f0cf0814700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:25.757 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.752+0000 7f0cf0814700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 0x7f0cd8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:25.758 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.756+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cdc0126f0 con 0x7f0cec0f6510 2023-11-15T06:02:25.963 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.962+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cec045ad0 con 0x7f0cec0f6510 2023-11-15T06:02:25.964 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.963+0000 7f0ce1ffb700 1 -- 172.21.15.137:0/748797208 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0cdc00f340 con 0x7f0cec0f6510 2023-11-15T06:02:25.964 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:25.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:25.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 msgr2=0x7f0cd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:25.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 0x7f0cd8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).stop 2023-11-15T06:02:25.966 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 msgr2=0x7f0cec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:25.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0cdc013510 tx=0x7f0cdc00b040).stop 2023-11-15T06:02:25.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 shutdown_connections 2023-11-15T06:02:25.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f0cd802ff60 0x7f0cd8032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:25.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 --2- 172.21.15.137:0/748797208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cec0f6510 0x7f0cec06b9a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:25.967 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 >> 172.21.15.137:0/748797208 conn(0x7f0cec0f1c70 msgr2=0x7f0cec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:25.968 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 shutdown_connections 2023-11-15T06:02:25.968 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:25.964+0000 7f0cf3279700 1 -- 172.21.15.137:0/748797208 wait complete. 2023-11-15T06:02:25.968 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:26.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:25.039384+0000 mgr.smithi003.cldzke (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:25.963929+0000 mon.smithi003 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.137:0/748797208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:27.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:27.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:28.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:27.039746+0000 mgr.smithi003.cldzke (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:28.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 -- 172.21.15.137:0/487227723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 msgr2=0x7f47f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:28.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 --2- 172.21.15.137:0/487227723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f00f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f47e0009fb0 tx=0x7f47e0009b30).stop 2023-11-15T06:02:28.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 -- 172.21.15.137:0/487227723 shutdown_connections 2023-11-15T06:02:28.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 --2- 172.21.15.137:0/487227723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f00f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:28.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 -- 172.21.15.137:0/487227723 >> 172.21.15.137:0/487227723 conn(0x7f47f00f0b70 msgr2=0x7f47f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:28.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 -- 172.21.15.137:0/487227723 shutdown_connections 2023-11-15T06:02:28.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.552+0000 7f47f499c700 1 -- 172.21.15.137:0/487227723 wait complete. 2023-11-15T06:02:28.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47f499c700 1 Processor -- start 2023-11-15T06:02:28.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47f499c700 1 -- start start 2023-11-15T06:02:28.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47f499c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:28.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47f499c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0060220 con 0x7f47f00f5410 2023-11-15T06:02:28.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:28.557 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47ee59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:39500/0 (socket says 172.21.15.137:39500) 2023-11-15T06:02:28.558 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.553+0000 7f47ee59c700 1 -- 172.21.15.137:0/233123870 learned_addr learned my addr 172.21.15.137:0/233123870 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:28.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47ee59c700 1 -- 172.21.15.137:0/233123870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e0005040 con 0x7f47f00f5410 2023-11-15T06:02:28.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47ee59c700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f47e0013510 tx=0x7f47e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:28.559 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e0009320 con 0x7f47f00f5410 2023-11-15T06:02:28.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f0060420 con 0x7f47f00f5410 2023-11-15T06:02:28.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47e0013f00 con 0x7f47f00f5410 2023-11-15T06:02:28.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e0014d90 con 0x7f47f00f5410 2023-11-15T06:02:28.560 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.554+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f0109b00 con 0x7f47f00f5410 2023-11-15T06:02:28.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47e00095f0 con 0x7f47f00f5410 2023-11-15T06:02:28.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0059100 con 0x7f47f00f5410 2023-11-15T06:02:28.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47deffd700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:28.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47e0040fe0 con 0x7f47f00f5410 2023-11-15T06:02:28.561 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47edd9b700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:28.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.555+0000 7f47edd9b700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 0x7f47d8032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f47e40096a0 tx=0x7f47e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:28.562 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.559+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47e001d7f0 con 0x7f47f00f5410 2023-11-15T06:02:28.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.766+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47f0045ad0 con 0x7f47f00f5410 2023-11-15T06:02:28.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.767+0000 7f47deffd700 1 -- 172.21.15.137:0/233123870 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f47e0010230 con 0x7f47f00f5410 2023-11-15T06:02:28.769 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:28.769 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:28.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 msgr2=0x7f47d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:28.770 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 0x7f47d8032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f47e40096a0 tx=0x7f47e4006b40).stop 2023-11-15T06:02:28.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 msgr2=0x7f47f0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:28.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f47e0013510 tx=0x7f47e000b040).stop 2023-11-15T06:02:28.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 shutdown_connections 2023-11-15T06:02:28.771 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.768+0000 7f47f499c700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f47d802ff60 0x7f47d8032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.769+0000 7f47f499c700 1 --2- 172.21.15.137:0/233123870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f00f5410 0x7f47f0109450 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.769+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 >> 172.21.15.137:0/233123870 conn(0x7f47f00f0b70 msgr2=0x7f47f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.769+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 shutdown_connections 2023-11-15T06:02:28.772 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:28.769+0000 7f47f499c700 1 -- 172.21.15.137:0/233123870 wait complete. 2023-11-15T06:02:28.773 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:29.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:29 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:28.768190+0000 mon.smithi003 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.137:0/233123870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:30.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:30.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:29.040024+0000 mgr.smithi003.cldzke (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:31.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- 172.21.15.137:0/3406580935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 msgr2=0x7f1d500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:31.320 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 --2- 172.21.15.137:0/3406580935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d500f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f1d44009fb0 tx=0x7f1d44009b30).stop 2023-11-15T06:02:31.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- 172.21.15.137:0/3406580935 shutdown_connections 2023-11-15T06:02:31.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 --2- 172.21.15.137:0/3406580935 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d500f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:31.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- 172.21.15.137:0/3406580935 >> 172.21.15.137:0/3406580935 conn(0x7f1d500f0b70 msgr2=0x7f1d500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:31.321 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- 172.21.15.137:0/3406580935 shutdown_connections 2023-11-15T06:02:31.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- 172.21.15.137:0/3406580935 wait complete. 2023-11-15T06:02:31.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 Processor -- start 2023-11-15T06:02:31.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.318+0000 7f1d56750700 1 -- start start 2023-11-15T06:02:31.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d56750700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:31.322 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d56750700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d50106c00 con 0x7f1d500f5410 2023-11-15T06:02:31.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:31.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:43908/0 (socket says 172.21.15.137:43908) 2023-11-15T06:02:31.324 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4ffff700 1 -- 172.21.15.137:0/2215223319 learned_addr learned my addr 172.21.15.137:0/2215223319 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4ffff700 1 -- 172.21.15.137:0/2215223319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44005040 con 0x7f1d500f5410 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4ffff700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f1d44013510 tx=0x7f1d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d44009320 con 0x7f1d500f5410 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.319+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d50106e00 con 0x7f1d500f5410 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.320+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d44013f00 con 0x7f1d500f5410 2023-11-15T06:02:31.325 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.320+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d44014d90 con 0x7f1d500f5410 2023-11-15T06:02:31.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.320+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d50107290 con 0x7f1d500f5410 2023-11-15T06:02:31.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.320+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d440095f0 con 0x7f1d500f5410 2023-11-15T06:02:31.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.320+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50045ad0 con 0x7f1d500f5410 2023-11-15T06:02:31.326 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.321+0000 7f1d4cff9700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 0x7f1d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:31.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.321+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d44041000 con 0x7f1d500f5410 2023-11-15T06:02:31.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.321+0000 7f1d4f7fe700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 0x7f1d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:31.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.321+0000 7f1d4f7fe700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 0x7f1d38032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1d400096a0 tx=0x7f1d40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:31.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.325+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d44014420 con 0x7f1d500f5410 2023-11-15T06:02:31.533 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.531+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d500fae60 con 0x7f1d500f5410 2023-11-15T06:02:31.534 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.532+0000 7f1d4cff9700 1 -- 172.21.15.137:0/2215223319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1d44010230 con 0x7f1d500f5410 2023-11-15T06:02:31.534 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:31.534 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:31.535 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 msgr2=0x7f1d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 0x7f1d38032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1d400096a0 tx=0x7f1d40006b40).stop 2023-11-15T06:02:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 msgr2=0x7f1d50106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f1d44013510 tx=0x7f1d4400b040).stop 2023-11-15T06:02:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 shutdown_connections 2023-11-15T06:02:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1d3802ff60 0x7f1d38032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:31.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 --2- 172.21.15.137:0/2215223319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d500f5410 0x7f1d50106710 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:31.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.533+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 >> 172.21.15.137:0/2215223319 conn(0x7f1d500f0b70 msgr2=0x7f1d5005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:31.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.534+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 shutdown_connections 2023-11-15T06:02:31.537 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:31.534+0000 7f1d56750700 1 -- 172.21.15.137:0/2215223319 wait complete. 2023-11-15T06:02:31.538 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:31.040393+0000 mgr.smithi003.cldzke (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:31.533397+0000 mon.smithi003 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.137:0/2215223319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:32.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:32.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:34.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 -- 172.21.15.137:0/2292805180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 msgr2=0x7fd5500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:34.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 --2- 172.21.15.137:0/2292805180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd5500f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fd538009fb0 tx=0x7fd538009b30).stop 2023-11-15T06:02:34.115 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 -- 172.21.15.137:0/2292805180 shutdown_connections 2023-11-15T06:02:34.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 --2- 172.21.15.137:0/2292805180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd5500f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:34.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 -- 172.21.15.137:0/2292805180 >> 172.21.15.137:0/2292805180 conn(0x7fd5500f1c70 msgr2=0x7fd5500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:34.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 -- 172.21.15.137:0/2292805180 shutdown_connections 2023-11-15T06:02:34.116 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.113+0000 7fd555039700 1 -- 172.21.15.137:0/2292805180 wait complete. 2023-11-15T06:02:34.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd555039700 1 Processor -- start 2023-11-15T06:02:34.118 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd555039700 1 -- start start 2023-11-15T06:02:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd555039700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd555039700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550106c20 con 0x7fd5500f6510 2023-11-15T06:02:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd54ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd54ed9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:43928/0 (socket says 172.21.15.137:43928) 2023-11-15T06:02:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd54ed9d700 1 -- 172.21.15.137:0/3610785069 learned_addr learned my addr 172.21.15.137:0/3610785069 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.114+0000 7fd54ed9d700 1 -- 172.21.15.137:0/3610785069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd538005040 con 0x7fd5500f6510 2023-11-15T06:02:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd54ed9d700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fd538013510 tx=0x7fd53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd538009320 con 0x7fd5500f6510 2023-11-15T06:02:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd538013f00 con 0x7fd5500f6510 2023-11-15T06:02:34.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd550106e20 con 0x7fd5500f6510 2023-11-15T06:02:34.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd538014d90 con 0x7fd5500f6510 2023-11-15T06:02:34.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.115+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5501072b0 con 0x7fd5500f6510 2023-11-15T06:02:34.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550045ad0 con 0x7fd5500f6510 2023-11-15T06:02:34.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5380095f0 con 0x7fd5500f6510 2023-11-15T06:02:34.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd5477fe700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 0x7fd53c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:34.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd538040fe0 con 0x7fd5500f6510 2023-11-15T06:02:34.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd54e59c700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 0x7fd53c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:34.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.116+0000 7fd54e59c700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 0x7fd53c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd5400096a0 tx=0x7fd540006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:34.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.121+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd53801d7f0 con 0x7fd5500f6510 2023-11-15T06:02:34.327 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.325+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5500fbf60 con 0x7fd5500f6510 2023-11-15T06:02:34.328 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.325+0000 7fd5477fe700 1 -- 172.21.15.137:0/3610785069 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd538010230 con 0x7fd5500f6510 2023-11-15T06:02:34.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:34.328 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:34.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 msgr2=0x7fd53c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:34.329 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 0x7fd53c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd5400096a0 tx=0x7fd540006b40).stop 2023-11-15T06:02:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 msgr2=0x7fd550106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fd538013510 tx=0x7fd53800b040).stop 2023-11-15T06:02:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 shutdown_connections 2023-11-15T06:02:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fd53c02ff60 0x7fd53c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 --2- 172.21.15.137:0/3610785069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5500f6510 0x7fd550106730 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:34.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 >> 172.21.15.137:0/3610785069 conn(0x7fd5500f1c70 msgr2=0x7fd5500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:34.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 shutdown_connections 2023-11-15T06:02:34.331 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:34.327+0000 7fd555039700 1 -- 172.21.15.137:0/3610785069 wait complete. 2023-11-15T06:02:34.331 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:34.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:34 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:33.040774+0000 mgr.smithi003.cldzke (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:35.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:35 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:34.327063+0000 mon.smithi003 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.137:0/3610785069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:35.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:35.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:35.041042+0000 mgr.smithi003.cldzke (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:35.117853+0000 mon.smithi003 (mon.0) 302 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:02:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:35.120330+0000 mon.smithi003 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T06:02:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 -- 172.21.15.137:0/3145531185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 msgr2=0x7ff8fc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 --2- 172.21.15.137:0/3145531185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc0f4be0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7ff8f0009fb0 tx=0x7ff8f0009b30).stop 2023-11-15T06:02:36.885 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 -- 172.21.15.137:0/3145531185 shutdown_connections 2023-11-15T06:02:36.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 --2- 172.21.15.137:0/3145531185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc0f4be0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:36.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 -- 172.21.15.137:0/3145531185 >> 172.21.15.137:0/3145531185 conn(0x7ff8fc0f02e0 msgr2=0x7ff8fc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:36.886 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 -- 172.21.15.137:0/3145531185 shutdown_connections 2023-11-15T06:02:36.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.882+0000 7ff901888700 1 -- 172.21.15.137:0/3145531185 wait complete. 2023-11-15T06:02:36.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.883+0000 7ff901888700 1 Processor -- start 2023-11-15T06:02:36.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.883+0000 7ff901888700 1 -- start start 2023-11-15T06:02:36.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.883+0000 7ff901888700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:36.887 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.883+0000 7ff901888700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc106bf0 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.883+0000 7ff8faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:36.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:43946/0 (socket says 172.21.15.137:43946) 2023-11-15T06:02:36.888 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8faffd700 1 -- 172.21.15.137:0/916469826 learned_addr learned my addr 172.21.15.137:0/916469826 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:36.889 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8faffd700 1 -- 172.21.15.137:0/916469826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0005040 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8faffd700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff8f0013510 tx=0x7ff8f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:36.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f0009320 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc106df0 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8f0013f00 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.890 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f0014d90 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.884+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc107280 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.885+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8f00095f0 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.885+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc045ad0 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.885+0000 7ff8e37fe700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:36.891 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.885+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8f0041000 con 0x7ff8fc0f47c0 2023-11-15T06:02:36.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.885+0000 7ff8fa7fc700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:36.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.886+0000 7ff8fa7fc700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff8ec0096a0 tx=0x7ff8ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:36.892 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:36.890+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8f0014420 con 0x7ff8fc0f47c0 2023-11-15T06:02:37.097 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.095+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8fc108130 con 0x7ff8fc0f47c0 2023-11-15T06:02:37.099 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff8e37fe700 1 -- 172.21.15.137:0/916469826 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff8f0010230 con 0x7ff8fc0f47c0 2023-11-15T06:02:37.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:37.100 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 msgr2=0x7ff8e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff8ec0096a0 tx=0x7ff8ec006b40).stop 2023-11-15T06:02:37.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 msgr2=0x7ff8fc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:37.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff8f0013510 tx=0x7ff8f000b040).stop 2023-11-15T06:02:37.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 shutdown_connections 2023-11-15T06:02:37.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff8e402ff60 0x7ff8e4032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:37.102 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 --2- 172.21.15.137:0/916469826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8fc0f47c0 0x7ff8fc106700 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:37.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.099+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 >> 172.21.15.137:0/916469826 conn(0x7ff8fc0f02e0 msgr2=0x7ff8fc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:37.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.100+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 shutdown_connections 2023-11-15T06:02:37.103 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:37.100+0000 7ff901888700 1 -- 172.21.15.137:0/916469826 wait complete. 2023-11-15T06:02:37.103 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:37.041259+0000 mgr.smithi003.cldzke (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:37.099442+0000 mon.smithi003 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.137:0/916469826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:38.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:38.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:39.682 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 -- 172.21.15.137:0/2703421926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 msgr2=0x7f16f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:39.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 --2- 172.21.15.137:0/2703421926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f80f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f16e8009fb0 tx=0x7f16e8009b30).stop 2023-11-15T06:02:39.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 -- 172.21.15.137:0/2703421926 shutdown_connections 2023-11-15T06:02:39.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 --2- 172.21.15.137:0/2703421926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f80f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:39.683 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 -- 172.21.15.137:0/2703421926 >> 172.21.15.137:0/2703421926 conn(0x7f16f80f0b70 msgr2=0x7f16f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:39.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 -- 172.21.15.137:0/2703421926 shutdown_connections 2023-11-15T06:02:39.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.680+0000 7f170056f700 1 -- 172.21.15.137:0/2703421926 wait complete. 2023-11-15T06:02:39.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f170056f700 1 Processor -- start 2023-11-15T06:02:39.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f170056f700 1 -- start start 2023-11-15T06:02:39.684 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f170056f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:39.685 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f170056f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8060220 con 0x7f16f80f5410 2023-11-15T06:02:39.686 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f16fe30b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:39.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f16fe30b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54666/0 (socket says 172.21.15.137:54666) 2023-11-15T06:02:39.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.681+0000 7f16fe30b700 1 -- 172.21.15.137:0/2341331505 learned_addr learned my addr 172.21.15.137:0/2341331505 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:39.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f16fe30b700 1 -- 172.21.15.137:0/2341331505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e8005040 con 0x7f16f80f5410 2023-11-15T06:02:39.687 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f16fe30b700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f16e8013510 tx=0x7f16e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:39.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e8009320 con 0x7f16f80f5410 2023-11-15T06:02:39.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f8060420 con 0x7f16f80f5410 2023-11-15T06:02:39.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16e8013f00 con 0x7f16f80f5410 2023-11-15T06:02:39.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e8014d90 con 0x7f16f80f5410 2023-11-15T06:02:39.688 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.682+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8109b00 con 0x7f16f80f5410 2023-11-15T06:02:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.683+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8059100 con 0x7f16f80f5410 2023-11-15T06:02:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.683+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16e80095f0 con 0x7f16f80f5410 2023-11-15T06:02:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.683+0000 7f16eeffd700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 0x7f16e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.683+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16e8041000 con 0x7f16f80f5410 2023-11-15T06:02:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.683+0000 7f16fdb0a700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 0x7f16e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:39.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.684+0000 7f16fdb0a700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 0x7f16e4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f16f40096a0 tx=0x7f16f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:39.690 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.688+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16e8014420 con 0x7f16f80f5410 2023-11-15T06:02:39.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.895+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16f8045ad0 con 0x7f16f80f5410 2023-11-15T06:02:39.898 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.896+0000 7f16eeffd700 1 -- 172.21.15.137:0/2341331505 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f16e8010230 con 0x7f16f80f5410 2023-11-15T06:02:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:39.899 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:39.899 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 msgr2=0x7f16e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:39.900 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 0x7f16e4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f16f40096a0 tx=0x7f16f4006b40).stop 2023-11-15T06:02:39.900 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 msgr2=0x7f16f8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:39.900 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f16e8013510 tx=0x7f16e800b040).stop 2023-11-15T06:02:39.900 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 shutdown_connections 2023-11-15T06:02:39.901 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f16e402ff60 0x7f16e4032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:39.901 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 --2- 172.21.15.137:0/2341331505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16f80f5410 0x7f16f8109450 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:39.901 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 >> 172.21.15.137:0/2341331505 conn(0x7f16f80f0b70 msgr2=0x7f16f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:39.901 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 shutdown_connections 2023-11-15T06:02:39.901 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:39.898+0000 7f170056f700 1 -- 172.21.15.137:0/2341331505 wait complete. 2023-11-15T06:02:39.902 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:39.041632+0000 mgr.smithi003.cldzke (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:40.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:40 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:39.897474+0000 mon.smithi003 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.137:0/2341331505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:41.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:41.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:41.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:41 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:41.041945+0000 mgr.smithi003.cldzke (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:42.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.455+0000 7fed48418700 1 -- 172.21.15.137:0/1681336014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 msgr2=0x7fed400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:42.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.455+0000 7fed48418700 1 --2- 172.21.15.137:0/1681336014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed400f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fed30009fb0 tx=0x7fed30009b30).stop 2023-11-15T06:02:42.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- 172.21.15.137:0/1681336014 shutdown_connections 2023-11-15T06:02:42.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 --2- 172.21.15.137:0/1681336014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed400f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:42.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- 172.21.15.137:0/1681336014 >> 172.21.15.137:0/1681336014 conn(0x7fed400f1c70 msgr2=0x7fed400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:42.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- 172.21.15.137:0/1681336014 shutdown_connections 2023-11-15T06:02:42.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- 172.21.15.137:0/1681336014 wait complete. 2023-11-15T06:02:42.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 Processor -- start 2023-11-15T06:02:42.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- start start 2023-11-15T06:02:42.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:42.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.456+0000 7fed48418700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed40106c20 con 0x7fed400f6510 2023-11-15T06:02:42.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed461b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:42.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed461b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54682/0 (socket says 172.21.15.137:54682) 2023-11-15T06:02:42.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed461b4700 1 -- 172.21.15.137:0/2224315704 learned_addr learned my addr 172.21.15.137:0/2224315704 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:42.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed461b4700 1 -- 172.21.15.137:0/2224315704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30005040 con 0x7fed400f6510 2023-11-15T06:02:42.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed461b4700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fed30013510 tx=0x7fed3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed30009320 con 0x7fed400f6510 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed40106e20 con 0x7fed400f6510 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed30013f00 con 0x7fed400f6510 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed30014d90 con 0x7fed400f6510 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.457+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed401072b0 con 0x7fed400f6510 2023-11-15T06:02:42.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.458+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed300095f0 con 0x7fed400f6510 2023-11-15T06:02:42.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.458+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed40045ad0 con 0x7fed400f6510 2023-11-15T06:02:42.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.458+0000 7fed36ffd700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 0x7fed2c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:42.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.458+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed300410c0 con 0x7fed400f6510 2023-11-15T06:02:42.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.458+0000 7fed459b3700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 0x7fed2c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:42.464 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.459+0000 7fed459b3700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 0x7fed2c032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fed3c0096a0 tx=0x7fed3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:42.465 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.463+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed30014420 con 0x7fed400f6510 2023-11-15T06:02:42.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.668+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed400fbf60 con 0x7fed400f6510 2023-11-15T06:02:42.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.669+0000 7fed36ffd700 1 -- 172.21.15.137:0/2224315704 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fed30010230 con 0x7fed400f6510 2023-11-15T06:02:42.674 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:42.674 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:42.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.670+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 msgr2=0x7fed2c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:42.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.670+0000 7fed48418700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 0x7fed2c032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fed3c0096a0 tx=0x7fed3c006b40).stop 2023-11-15T06:02:42.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.670+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 msgr2=0x7fed40106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:42.675 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.670+0000 7fed48418700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fed30013510 tx=0x7fed3000b040).stop 2023-11-15T06:02:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.670+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 shutdown_connections 2023-11-15T06:02:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.671+0000 7fed48418700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fed2c02ffb0 0x7fed2c032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.671+0000 7fed48418700 1 --2- 172.21.15.137:0/2224315704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed400f6510 0x7fed40106730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.671+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 >> 172.21.15.137:0/2224315704 conn(0x7fed400f1c70 msgr2=0x7fed400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.671+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 shutdown_connections 2023-11-15T06:02:42.677 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:42.671+0000 7fed48418700 1 -- 172.21.15.137:0/2224315704 wait complete. 2023-11-15T06:02:42.677 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:43.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:42 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:42.670167+0000 mon.smithi003 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.137:0/2224315704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:44.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:44.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:44.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:43 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:43.042335+0000 mgr.smithi003.cldzke (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:45.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- 172.21.15.137:0/2327442018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 msgr2=0x7f3f6c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:45.215 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/2327442018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c0f8f20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f3f60009fb0 tx=0x7f3f60009b30).stop 2023-11-15T06:02:45.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- 172.21.15.137:0/2327442018 shutdown_connections 2023-11-15T06:02:45.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/2327442018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c0f8f20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:45.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- 172.21.15.137:0/2327442018 >> 172.21.15.137:0/2327442018 conn(0x7f3f6c0f09a0 msgr2=0x7f3f6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:45.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- 172.21.15.137:0/2327442018 shutdown_connections 2023-11-15T06:02:45.216 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- 172.21.15.137:0/2327442018 wait complete. 2023-11-15T06:02:45.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 Processor -- start 2023-11-15T06:02:45.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.213+0000 7f3f6bfff700 1 -- start start 2023-11-15T06:02:45.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:45.217 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c1069f0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:45.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54706/0 (socket says 172.21.15.137:54706) 2023-11-15T06:02:45.219 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6affd700 1 -- 172.21.15.137:0/789235454 learned_addr learned my addr 172.21.15.137:0/789235454 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6affd700 1 -- 172.21.15.137:0/789235454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f60005040 con 0x7f3f6c0601e0 2023-11-15T06:02:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.214+0000 7f3f6affd700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3f60013510 tx=0x7f3f6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f60009320 con 0x7f3f6c0601e0 2023-11-15T06:02:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c106bf0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f60013f00 con 0x7f3f6c0601e0 2023-11-15T06:02:45.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f60014d90 con 0x7f3f6c0601e0 2023-11-15T06:02:45.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c107080 con 0x7f3f6c0601e0 2023-11-15T06:02:45.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f600095f0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.215+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c045ad0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.221 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.216+0000 7f3f537fe700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 0x7f3f54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.216+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f600410a0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.216+0000 7f3f6a7fc700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 0x7f3f54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.216+0000 7f3f6a7fc700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 0x7f3f54032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3f5c0096a0 tx=0x7f3f5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.221+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f60010230 con 0x7f3f6c0601e0 2023-11-15T06:02:45.428 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.427+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f6c107ed0 con 0x7f3f6c0601e0 2023-11-15T06:02:45.429 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.428+0000 7f3f537fe700 1 -- 172.21.15.137:0/789235454 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3f60010230 con 0x7f3f6c0601e0 2023-11-15T06:02:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:45.430 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:45.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 msgr2=0x7f3f54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:45.431 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 0x7f3f54032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3f5c0096a0 tx=0x7f3f5c006b40).stop 2023-11-15T06:02:45.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 msgr2=0x7f3f6c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:45.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3f60013510 tx=0x7f3f6000b040).stop 2023-11-15T06:02:45.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 shutdown_connections 2023-11-15T06:02:45.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f3f5402ff60 0x7f3f54032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:45.432 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 --2- 172.21.15.137:0/789235454 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f6c0601e0 0x7f3f6c106500 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:45.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 >> 172.21.15.137:0/789235454 conn(0x7f3f6c0f09a0 msgr2=0x7f3f6c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:45.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 shutdown_connections 2023-11-15T06:02:45.433 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:45.429+0000 7f3f6bfff700 1 -- 172.21.15.137:0/789235454 wait complete. 2023-11-15T06:02:45.433 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:45.042687+0000 mgr.smithi003.cldzke (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:46.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:46 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:45.428973+0000 mon.smithi003 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.137:0/789235454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:46.849 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:46.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:48.054 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.052+0000 7f1c5f449700 1 -- 172.21.15.137:0/619269987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 msgr2=0x7f1c580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:48.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.052+0000 7f1c5f449700 1 --2- 172.21.15.137:0/619269987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c580f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f1c54009fb0 tx=0x7f1c54009b30).stop 2023-11-15T06:02:48.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 -- 172.21.15.137:0/619269987 shutdown_connections 2023-11-15T06:02:48.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 --2- 172.21.15.137:0/619269987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c580f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:48.055 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 -- 172.21.15.137:0/619269987 >> 172.21.15.137:0/619269987 conn(0x7f1c580f1c70 msgr2=0x7f1c580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:48.056 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 -- 172.21.15.137:0/619269987 shutdown_connections 2023-11-15T06:02:48.056 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 -- 172.21.15.137:0/619269987 wait complete. 2023-11-15T06:02:48.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 Processor -- start 2023-11-15T06:02:48.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.053+0000 7f1c5f449700 1 -- start start 2023-11-15T06:02:48.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5f449700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:48.058 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5f449700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c58108dd0 con 0x7f1c580f6510 2023-11-15T06:02:48.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5d1e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:48.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5d1e5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54724/0 (socket says 172.21.15.137:54724) 2023-11-15T06:02:48.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5d1e5700 1 -- 172.21.15.137:0/542027318 learned_addr learned my addr 172.21.15.137:0/542027318 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:48.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5d1e5700 1 -- 172.21.15.137:0/542027318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c54005040 con 0x7f1c580f6510 2023-11-15T06:02:48.059 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.054+0000 7f1c5d1e5700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f1c54013510 tx=0x7f1c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:48.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.055+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c54009320 con 0x7f1c580f6510 2023-11-15T06:02:48.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.055+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c58108fd0 con 0x7f1c580f6510 2023-11-15T06:02:48.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.055+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c54013f00 con 0x7f1c580f6510 2023-11-15T06:02:48.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.055+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c54014d90 con 0x7f1c580f6510 2023-11-15T06:02:48.060 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.055+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c58109460 con 0x7f1c580f6510 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c540263a0 con 0x7f1c580f6510 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58045ad0 con 0x7f1c580f6510 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c4dffb700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 0x7f1c44032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c54041270 con 0x7f1c580f6510 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c5c9e4700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 0x7f1c44032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:48.061 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.056+0000 7f1c5c9e4700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 0x7f1c44032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1c480096a0 tx=0x7f1c48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:48.062 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.061+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c54012510 con 0x7f1c580f6510 2023-11-15T06:02:48.265 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.264+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c5810aa80 con 0x7f1c580f6510 2023-11-15T06:02:48.266 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.265+0000 7f1c4dffb700 1 -- 172.21.15.137:0/542027318 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1c5400f340 con 0x7f1c580f6510 2023-11-15T06:02:48.266 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:48.267 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:48.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 msgr2=0x7f1c44032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:48.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 0x7f1c44032470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f1c480096a0 tx=0x7f1c48006b40).stop 2023-11-15T06:02:48.268 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 msgr2=0x7f1c581088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f1c54013510 tx=0x7f1c5400b040).stop 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 shutdown_connections 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f1c4402ffb0 0x7f1c44032470 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 --2- 172.21.15.137:0/542027318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c580f6510 0x7f1c581088e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 >> 172.21.15.137:0/542027318 conn(0x7f1c580f1c70 msgr2=0x7f1c580f4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 shutdown_connections 2023-11-15T06:02:48.270 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:48.266+0000 7f1c5f449700 1 -- 172.21.15.137:0/542027318 wait complete. 2023-11-15T06:02:48.270 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:48.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:48 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:47.043021+0000 mgr.smithi003.cldzke (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:49.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:49 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:48.266124+0000 mon.smithi003 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.137:0/542027318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:49.646 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:49.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:50.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:50 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:49.043387+0000 mgr.smithi003.cldzke (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:50.759 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 -- 172.21.15.137:0/646592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 msgr2=0x7f36e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:50.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/646592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e00f6930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f36d4009fb0 tx=0x7f36d4009b30).stop 2023-11-15T06:02:50.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 -- 172.21.15.137:0/646592671 shutdown_connections 2023-11-15T06:02:50.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/646592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e00f6930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:50.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 -- 172.21.15.137:0/646592671 >> 172.21.15.137:0/646592671 conn(0x7f36e00f1c70 msgr2=0x7f36e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:50.760 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 -- 172.21.15.137:0/646592671 shutdown_connections 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.756+0000 7f36e4f4a700 1 -- 172.21.15.137:0/646592671 wait complete. 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36e4f4a700 1 Processor -- start 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36e4f4a700 1 -- start start 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36e4f4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36e4f4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e0106c20 con 0x7f36e00f6510 2023-11-15T06:02:50.761 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54928/0 (socket says 172.21.15.137:54928) 2023-11-15T06:02:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36de59c700 1 -- 172.21.15.137:0/1453917149 learned_addr learned my addr 172.21.15.137:0/1453917149 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36de59c700 1 -- 172.21.15.137:0/1453917149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4005040 con 0x7f36e00f6510 2023-11-15T06:02:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.757+0000 7f36de59c700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f36d4013510 tx=0x7f36d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4009320 con 0x7f36e00f6510 2023-11-15T06:02:50.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e0106e20 con 0x7f36e00f6510 2023-11-15T06:02:50.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36d4013f00 con 0x7f36e00f6510 2023-11-15T06:02:50.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36d4014d90 con 0x7f36e00f6510 2023-11-15T06:02:50.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e01072b0 con 0x7f36e00f6510 2023-11-15T06:02:50.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36d40095f0 con 0x7f36e00f6510 2023-11-15T06:02:50.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.758+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e0045ad0 con 0x7f36e00f6510 2023-11-15T06:02:50.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.759+0000 7f36ceffd700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 0x7f36c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:50.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.759+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f36d4040fc0 con 0x7f36e00f6510 2023-11-15T06:02:50.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.759+0000 7f36ddd9b700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 0x7f36c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:50.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.759+0000 7f36ddd9b700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 0x7f36c8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f36d00096a0 tx=0x7f36d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:50.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.764+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36d4014420 con 0x7f36e00f6510 2023-11-15T06:02:50.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.972+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36e00fbf60 con 0x7f36e00f6510 2023-11-15T06:02:50.974 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.972+0000 7f36ceffd700 1 -- 172.21.15.137:0/1453917149 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f36d4010230 con 0x7f36e00f6510 2023-11-15T06:02:50.975 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:50.975 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:50.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 msgr2=0x7f36c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:50.976 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 0x7f36c8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f36d00096a0 tx=0x7f36d0006b40).stop 2023-11-15T06:02:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 msgr2=0x7f36e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f36d4013510 tx=0x7f36d400b040).stop 2023-11-15T06:02:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 shutdown_connections 2023-11-15T06:02:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f36c802ff60 0x7f36c8032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 --2- 172.21.15.137:0/1453917149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36e00f6510 0x7f36e0106730 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:50.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 >> 172.21.15.137:0/1453917149 conn(0x7f36e00f1c70 msgr2=0x7f36e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:50.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 shutdown_connections 2023-11-15T06:02:50.978 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:50.974+0000 7f36e4f4a700 1 -- 172.21.15.137:0/1453917149 wait complete. 2023-11-15T06:02:50.978 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:50.973934+0000 mon.smithi003 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.137:0/1453917149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:51.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:51 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:51.043691+0000 mgr.smithi003.cldzke (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:52.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:52.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:53.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 -- 172.21.15.137:0/2942670591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 msgr2=0x7fc5640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:53.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 --2- 172.21.15.137:0/2942670591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5640f55c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fc554009fb0 tx=0x7fc554009b30).stop 2023-11-15T06:02:53.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 -- 172.21.15.137:0/2942670591 shutdown_connections 2023-11-15T06:02:53.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 --2- 172.21.15.137:0/2942670591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5640f55c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:53.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 -- 172.21.15.137:0/2942670591 >> 172.21.15.137:0/2942670591 conn(0x7fc5640f09a0 msgr2=0x7fc5640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:53.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 -- 172.21.15.137:0/2942670591 shutdown_connections 2023-11-15T06:02:53.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 -- 172.21.15.137:0/2942670591 wait complete. 2023-11-15T06:02:53.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.545+0000 7fc56359e700 1 Processor -- start 2023-11-15T06:02:53.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56359e700 1 -- start start 2023-11-15T06:02:53.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:53.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5641069d0 con 0x7fc5640f51a0 2023-11-15T06:02:53.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:53.553 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54942/0 (socket says 172.21.15.137:54942) 2023-11-15T06:02:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56259c700 1 -- 172.21.15.137:0/1486754493 learned_addr learned my addr 172.21.15.137:0/1486754493 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.546+0000 7fc56259c700 1 -- 172.21.15.137:0/1486754493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554005040 con 0x7fc5640f51a0 2023-11-15T06:02:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc56259c700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fc554013510 tx=0x7fc55400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc554009320 con 0x7fc5640f51a0 2023-11-15T06:02:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc564106bd0 con 0x7fc5640f51a0 2023-11-15T06:02:53.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc554013f00 con 0x7fc5640f51a0 2023-11-15T06:02:53.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc554014dc0 con 0x7fc5640f51a0 2023-11-15T06:02:53.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.547+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc564107060 con 0x7fc5640f51a0 2023-11-15T06:02:53.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5540095f0 con 0x7fc5640f51a0 2023-11-15T06:02:53.555 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564045ad0 con 0x7fc5640f51a0 2023-11-15T06:02:53.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc552ffd700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 0x7fc54c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:53.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc554041020 con 0x7fc5640f51a0 2023-11-15T06:02:53.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc561d9b700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 0x7fc54c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:53.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.548+0000 7fc561d9b700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 0x7fc54c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc5580096a0 tx=0x7fc558006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:53.556 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.553+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc554014420 con 0x7fc5640f51a0 2023-11-15T06:02:53.762 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.761+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc564107eb0 con 0x7fc5640f51a0 2023-11-15T06:02:53.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.762+0000 7fc552ffd700 1 -- 172.21.15.137:0/1486754493 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc55401c920 con 0x7fc5640f51a0 2023-11-15T06:02:53.764 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:53.764 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 msgr2=0x7fc54c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 0x7fc54c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc5580096a0 tx=0x7fc558006b40).stop 2023-11-15T06:02:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 msgr2=0x7fc5641064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fc554013510 tx=0x7fc55400b040).stop 2023-11-15T06:02:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 shutdown_connections 2023-11-15T06:02:53.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc54c02ff60 0x7fc54c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:53.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 --2- 172.21.15.137:0/1486754493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5640f51a0 0x7fc5641064e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:53.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 >> 172.21.15.137:0/1486754493 conn(0x7fc5640f09a0 msgr2=0x7fc56405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:53.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 shutdown_connections 2023-11-15T06:02:53.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:53.764+0000 7fc56359e700 1 -- 172.21.15.137:0/1486754493 wait complete. 2023-11-15T06:02:53.767 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:54.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:53.044055+0000 mgr.smithi003.cldzke (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:54.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:54 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:53.762776+0000 mon.smithi003 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.137:0/1486754493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:55.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:55.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:56.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 -- 172.21.15.137:0/3698021107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 msgr2=0x7f931c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:56.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 --2- 172.21.15.137:0/3698021107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c0f68a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f930c009fb0 tx=0x7f930c009b30).stop 2023-11-15T06:02:56.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 -- 172.21.15.137:0/3698021107 shutdown_connections 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 --2- 172.21.15.137:0/3698021107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c0f68a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 -- 172.21.15.137:0/3698021107 >> 172.21.15.137:0/3698021107 conn(0x7f931c0f1c00 msgr2=0x7f931c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 -- 172.21.15.137:0/3698021107 shutdown_connections 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 -- 172.21.15.137:0/3698021107 wait complete. 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.290+0000 7f932205d700 1 Processor -- start 2023-11-15T06:02:56.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f932205d700 1 -- start start 2023-11-15T06:02:56.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f932205d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:56.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f932205d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c069e20 con 0x7f931c0f6480 2023-11-15T06:02:56.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f931b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:56.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f931b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54958/0 (socket says 172.21.15.137:54958) 2023-11-15T06:02:56.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f931b7fe700 1 -- 172.21.15.137:0/988473705 learned_addr learned my addr 172.21.15.137:0/988473705 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:56.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f931b7fe700 1 -- 172.21.15.137:0/988473705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c005040 con 0x7f931c0f6480 2023-11-15T06:02:56.295 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.291+0000 7f931b7fe700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f930c013510 tx=0x7f930c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:56.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.292+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f930c009320 con 0x7f931c0f6480 2023-11-15T06:02:56.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.292+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c069fc0 con 0x7f931c0f6480 2023-11-15T06:02:56.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.292+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f930c013f00 con 0x7f931c0f6480 2023-11-15T06:02:56.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.292+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f930c014d90 con 0x7f931c0f6480 2023-11-15T06:02:56.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.292+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c06a450 con 0x7f931c0f6480 2023-11-15T06:02:56.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f930c0263a0 con 0x7f931c0f6480 2023-11-15T06:02:56.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f9303fff700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:56.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f930c0410d0 con 0x7f931c0f6480 2023-11-15T06:02:56.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c059100 con 0x7f931c0f6480 2023-11-15T06:02:56.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f931affd700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:56.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.293+0000 7f931affd700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f93100096a0 tx=0x7f9310006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:56.299 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.298+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f930c012920 con 0x7f931c0f6480 2023-11-15T06:02:56.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:56 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:55.044414+0000 mgr.smithi003.cldzke (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:56.507 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.507+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f931c06ab50 con 0x7f931c0f6480 2023-11-15T06:02:56.508 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.507+0000 7f9303fff700 1 -- 172.21.15.137:0/988473705 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f930c02b020 con 0x7f931c0f6480 2023-11-15T06:02:56.508 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:56.509 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:56.509 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 msgr2=0x7f9304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f93100096a0 tx=0x7f9310006b40).stop 2023-11-15T06:02:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 msgr2=0x7f931c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f930c013510 tx=0x7f930c00b040).stop 2023-11-15T06:02:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 shutdown_connections 2023-11-15T06:02:56.510 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 --2- 172.21.15.137:0/988473705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f931c0f6480 0x7f931c06b9a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 >> 172.21.15.137:0/988473705 conn(0x7f931c0f1c00 msgr2=0x7f931c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 shutdown_connections 2023-11-15T06:02:56.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:56.509+0000 7f932205d700 1 -- 172.21.15.137:0/988473705 wait complete. 2023-11-15T06:02:56.512 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:02:57.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:57 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:56.507680+0000 mon.smithi003 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.137:0/988473705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:02:57.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:02:57.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:02:58.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:02:58 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:57.044823+0000 mgr.smithi003.cldzke (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:02:59.041 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.039+0000 7fee01b89700 1 -- 172.21.15.137:0/3225073045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 msgr2=0x7fedfc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:59.041 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.039+0000 7fee01b89700 1 --2- 172.21.15.137:0/3225073045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc0f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fedf0009fb0 tx=0x7fedf0009b30).stop 2023-11-15T06:02:59.041 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 -- 172.21.15.137:0/3225073045 shutdown_connections 2023-11-15T06:02:59.041 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 --2- 172.21.15.137:0/3225073045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc0f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:59.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 -- 172.21.15.137:0/3225073045 >> 172.21.15.137:0/3225073045 conn(0x7fedfc0f1c70 msgr2=0x7fedfc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:59.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 -- 172.21.15.137:0/3225073045 shutdown_connections 2023-11-15T06:02:59.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 -- 172.21.15.137:0/3225073045 wait complete. 2023-11-15T06:02:59.042 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 Processor -- start 2023-11-15T06:02:59.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.040+0000 7fee01b89700 1 -- start start 2023-11-15T06:02:59.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fee01b89700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:59.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fee01b89700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc106c20 con 0x7fedfc0f6510 2023-11-15T06:02:59.044 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fedfb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fedfb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:54964/0 (socket says 172.21.15.137:54964) 2023-11-15T06:02:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fedfb7fe700 1 -- 172.21.15.137:0/3680118406 learned_addr learned my addr 172.21.15.137:0/3680118406 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:02:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fedfb7fe700 1 -- 172.21.15.137:0/3680118406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0005040 con 0x7fedfc0f6510 2023-11-15T06:02:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.041+0000 7fedfb7fe700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fedf0014000 tx=0x7fedf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.042+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf00095f0 con 0x7fedfc0f6510 2023-11-15T06:02:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.042+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc106e20 con 0x7fedfc0f6510 2023-11-15T06:02:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.042+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedf0007e70 con 0x7fedfc0f6510 2023-11-15T06:02:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.042+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf0014ea0 con 0x7fedfc0f6510 2023-11-15T06:02:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.042+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc1072b0 con 0x7fedfc0f6510 2023-11-15T06:02:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedf0025070 con 0x7fedfc0f6510 2023-11-15T06:02:59.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc045ad0 con 0x7fedfc0f6510 2023-11-15T06:02:59.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fede3fff700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:02:59.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedf0040a30 con 0x7fedfc0f6510 2023-11-15T06:02:59.047 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fedfaffd700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:02:59.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.043+0000 7fedfaffd700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 0x7fede4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fedec0096a0 tx=0x7fedec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:02:59.048 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.047+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedf00079c0 con 0x7fedfc0f6510 2023-11-15T06:02:59.254 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.254+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedfc0fbf60 con 0x7fedfc0f6510 2023-11-15T06:02:59.255 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.254+0000 7fede3fff700 1 -- 172.21.15.137:0/3680118406 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fedf000f300 con 0x7fedfc0f6510 2023-11-15T06:02:59.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:02:59.256 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:02:59.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 msgr2=0x7fede4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:59.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 0x7fede4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fedec0096a0 tx=0x7fedec006b40).stop 2023-11-15T06:02:59.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 msgr2=0x7fedfc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:02:59.258 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fedf0014000 tx=0x7fedf000b040).stop 2023-11-15T06:02:59.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 shutdown_connections 2023-11-15T06:02:59.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:59.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 --2- 172.21.15.137:0/3680118406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fedfc0f6510 0x7fedfc106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:02:59.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 >> 172.21.15.137:0/3680118406 conn(0x7fedfc0f1c70 msgr2=0x7fedfc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:02:59.259 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 shutdown_connections 2023-11-15T06:02:59.260 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:02:59.256+0000 7fee01b89700 1 -- 172.21.15.137:0/3680118406 wait complete. 2023-11-15T06:02:59.260 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:02:59.045036+0000 mgr.smithi003.cldzke (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:00.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:00 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:02:59.254974+0000 mon.smithi003 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.137:0/3680118406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:00.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:00.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:01.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 -- 172.21.15.137:0/429247607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 msgr2=0x7fa7b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:01.910 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/429247607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b80f66c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa7a8009fb0 tx=0x7fa7a8009b30).stop 2023-11-15T06:03:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 -- 172.21.15.137:0/429247607 shutdown_connections 2023-11-15T06:03:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/429247607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b80f66c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 -- 172.21.15.137:0/429247607 >> 172.21.15.137:0/429247607 conn(0x7fa7b80f1aa0 msgr2=0x7fa7b80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:01.911 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 -- 172.21.15.137:0/429247607 shutdown_connections 2023-11-15T06:03:01.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 -- 172.21.15.137:0/429247607 wait complete. 2023-11-15T06:03:01.912 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.907+0000 7fa7b7fff700 1 Processor -- start 2023-11-15T06:03:01.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b7fff700 1 -- start start 2023-11-15T06:03:01.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:01.913 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81069a0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:01.914 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42484/0 (socket says 172.21.15.137:42484) 2023-11-15T06:03:01.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b6ffd700 1 -- 172.21.15.137:0/900873979 learned_addr learned my addr 172.21.15.137:0/900873979 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:01.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b6ffd700 1 -- 172.21.15.137:0/900873979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8005040 con 0x7fa7b80f62a0 2023-11-15T06:03:01.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.908+0000 7fa7b6ffd700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa7a8013510 tx=0x7fa7a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:01.915 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a8009320 con 0x7fa7b80f62a0 2023-11-15T06:03:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7a8013f00 con 0x7fa7b80f62a0 2023-11-15T06:03:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b8106ba0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a8014dc0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b8107030 con 0x7fa7b80f62a0 2023-11-15T06:03:01.917 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.909+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7a801d3d0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.910+0000 7fa79f7fe700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:01.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.910+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b8045ad0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.910+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7a80410d0 con 0x7fa7b80f62a0 2023-11-15T06:03:01.918 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.910+0000 7fa7b67fc700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:01.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.910+0000 7fa7b67fc700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 0x7fa7a0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa7ac0096a0 tx=0x7fa7ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:01.919 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:01.915+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7a8012920 con 0x7fa7b80f62a0 2023-11-15T06:03:01.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:01 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:01.045384+0000 mgr.smithi003.cldzke (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:02.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.118+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7b8107ee0 con 0x7fa7b80f62a0 2023-11-15T06:03:02.119 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.119+0000 7fa79f7fe700 1 -- 172.21.15.137:0/900873979 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa7a8012330 con 0x7fa7b80f62a0 2023-11-15T06:03:02.120 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:02.120 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:02.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 msgr2=0x7fa7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:02.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 0x7fa7a0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa7ac0096a0 tx=0x7fa7ac006b40).stop 2023-11-15T06:03:02.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 msgr2=0x7fa7b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:02.121 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa7a8013510 tx=0x7fa7a800b040).stop 2023-11-15T06:03:02.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 shutdown_connections 2023-11-15T06:03:02.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:02.122 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 --2- 172.21.15.137:0/900873979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa7b80f62a0 0x7fa7b81064b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:02.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 >> 172.21.15.137:0/900873979 conn(0x7fa7b80f1aa0 msgr2=0x7fa7b80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:02.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 shutdown_connections 2023-11-15T06:03:02.123 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:02.120+0000 7fa7b7fff700 1 -- 172.21.15.137:0/900873979 wait complete. 2023-11-15T06:03:02.123 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:02.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:02 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:02.119292+0000 mon.smithi003 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.137:0/900873979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:03.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:03.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:03.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:03 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:03.045776+0000 mgr.smithi003.cldzke (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:04.662 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.661+0000 7f2631220700 1 -- 172.21.15.137:0/4262720852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 msgr2=0x7f262c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:04.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.661+0000 7f2631220700 1 --2- 172.21.15.137:0/4262720852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c0f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2620009fb0 tx=0x7f2620009b30).stop 2023-11-15T06:03:04.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.661+0000 7f2631220700 1 -- 172.21.15.137:0/4262720852 shutdown_connections 2023-11-15T06:03:04.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.661+0000 7f2631220700 1 --2- 172.21.15.137:0/4262720852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c0f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:04.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.661+0000 7f2631220700 1 -- 172.21.15.137:0/4262720852 >> 172.21.15.137:0/4262720852 conn(0x7f262c0f1c70 msgr2=0x7f262c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:04.663 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.662+0000 7f2631220700 1 -- 172.21.15.137:0/4262720852 shutdown_connections 2023-11-15T06:03:04.664 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.662+0000 7f2631220700 1 -- 172.21.15.137:0/4262720852 wait complete. 2023-11-15T06:03:04.665 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.662+0000 7f2631220700 1 Processor -- start 2023-11-15T06:03:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.662+0000 7f2631220700 1 -- start start 2023-11-15T06:03:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f2631220700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f2631220700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c106c20 con 0x7f262c0f6510 2023-11-15T06:03:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f262ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f262ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42504/0 (socket says 172.21.15.137:42504) 2023-11-15T06:03:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f262ad9d700 1 -- 172.21.15.137:0/3823648890 learned_addr learned my addr 172.21.15.137:0/3823648890 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f262ad9d700 1 -- 172.21.15.137:0/3823648890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2620005040 con 0x7f262c0f6510 2023-11-15T06:03:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.663+0000 7f262ad9d700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2620013510 tx=0x7f262000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2620009320 con 0x7f262c0f6510 2023-11-15T06:03:04.667 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c106e20 con 0x7f262c0f6510 2023-11-15T06:03:04.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2620013f00 con 0x7f262c0f6510 2023-11-15T06:03:04.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2620014d90 con 0x7f262c0f6510 2023-11-15T06:03:04.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c1072b0 con 0x7f262c0f6510 2023-11-15T06:03:04.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2620014420 con 0x7f262c0f6510 2023-11-15T06:03:04.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.664+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c045ad0 con 0x7f262c0f6510 2023-11-15T06:03:04.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.665+0000 7f26137fe700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 0x7f2614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:04.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.665+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2620041000 con 0x7f262c0f6510 2023-11-15T06:03:04.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.665+0000 7f262a59c700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 0x7f2614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:04.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.665+0000 7f262a59c700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 0x7f2614032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:04.670 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.669+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26200095f0 con 0x7f262c0f6510 2023-11-15T06:03:04.875 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.875+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f262c0fbf60 con 0x7f262c0f6510 2023-11-15T06:03:04.876 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.875+0000 7f26137fe700 1 -- 172.21.15.137:0/3823648890 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f262001db70 con 0x7f262c0f6510 2023-11-15T06:03:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:04.877 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:04.877 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 msgr2=0x7f2614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:04.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 0x7f2614032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).stop 2023-11-15T06:03:04.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 msgr2=0x7f262c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:04.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2620013510 tx=0x7f262000b040).stop 2023-11-15T06:03:04.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 shutdown_connections 2023-11-15T06:03:04.878 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f261402ff60 0x7f2614032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:04.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 --2- 172.21.15.137:0/3823648890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f262c0f6510 0x7f262c106730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:04.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 >> 172.21.15.137:0/3823648890 conn(0x7f262c0f1c70 msgr2=0x7f262c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:04.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 shutdown_connections 2023-11-15T06:03:04.879 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:04.877+0000 7f2631220700 1 -- 172.21.15.137:0/3823648890 wait complete. 2023-11-15T06:03:04.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:05.181 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:04 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:04.875757+0000 mon.smithi003 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.137:0/3823648890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:06.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:06.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:06.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:05 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:05.046117+0000 mgr.smithi003.cldzke (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:07.455 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 -- 172.21.15.137:0/2050454246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 msgr2=0x7fe0940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:07.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2050454246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe0940f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fe084009fb0 tx=0x7fe084009b30).stop 2023-11-15T06:03:07.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 -- 172.21.15.137:0/2050454246 shutdown_connections 2023-11-15T06:03:07.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2050454246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe0940f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:07.456 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 -- 172.21.15.137:0/2050454246 >> 172.21.15.137:0/2050454246 conn(0x7fe0940f1c70 msgr2=0x7fe0940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 -- 172.21.15.137:0/2050454246 shutdown_connections 2023-11-15T06:03:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.454+0000 7fe09a8da700 1 -- 172.21.15.137:0/2050454246 wait complete. 2023-11-15T06:03:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe09a8da700 1 Processor -- start 2023-11-15T06:03:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe09a8da700 1 -- start start 2023-11-15T06:03:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe09a8da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:07.458 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe09a8da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094106c20 con 0x7fe0940f6510 2023-11-15T06:03:07.459 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe093fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:07.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe093fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:42522/0 (socket says 172.21.15.137:42522) 2023-11-15T06:03:07.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe093fff700 1 -- 172.21.15.137:0/2181710149 learned_addr learned my addr 172.21.15.137:0/2181710149 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:07.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.455+0000 7fe093fff700 1 -- 172.21.15.137:0/2181710149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084005040 con 0x7fe0940f6510 2023-11-15T06:03:07.460 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe093fff700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe084013510 tx=0x7fe08400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:07.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe084009320 con 0x7fe0940f6510 2023-11-15T06:03:07.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe084013f00 con 0x7fe0940f6510 2023-11-15T06:03:07.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe094106e20 con 0x7fe0940f6510 2023-11-15T06:03:07.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe084014d90 con 0x7fe0940f6510 2023-11-15T06:03:07.461 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.456+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0941072b0 con 0x7fe0940f6510 2023-11-15T06:03:07.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.457+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0840095f0 con 0x7fe0940f6510 2023-11-15T06:03:07.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.457+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094045ad0 con 0x7fe0940f6510 2023-11-15T06:03:07.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.457+0000 7fe090ff9700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:07.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.457+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0840410c0 con 0x7fe0940f6510 2023-11-15T06:03:07.462 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.457+0000 7fe0937fe700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:07.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.458+0000 7fe0937fe700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 0x7fe07c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0880096a0 tx=0x7fe088006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:07.463 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.462+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe084014420 con 0x7fe0940f6510 2023-11-15T06:03:07.668 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.668+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0940fbf60 con 0x7fe0940f6510 2023-11-15T06:03:07.669 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.669+0000 7fe090ff9700 1 -- 172.21.15.137:0/2181710149 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe08401d7f0 con 0x7fe0940f6510 2023-11-15T06:03:07.670 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:07.670 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:07.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 msgr2=0x7fe07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:07.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 0x7fe07c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0880096a0 tx=0x7fe088006b40).stop 2023-11-15T06:03:07.671 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 msgr2=0x7fe094106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe084013510 tx=0x7fe08400b040).stop 2023-11-15T06:03:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 shutdown_connections 2023-11-15T06:03:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe07c02ff60 0x7fe07c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 --2- 172.21.15.137:0/2181710149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0940f6510 0x7fe094106730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 >> 172.21.15.137:0/2181710149 conn(0x7fe0940f1c70 msgr2=0x7fe0940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:07.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 shutdown_connections 2023-11-15T06:03:07.673 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:07.670+0000 7fe09a8da700 1 -- 172.21.15.137:0/2181710149 wait complete. 2023-11-15T06:03:07.673 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:07.046516+0000 mgr.smithi003.cldzke (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:08.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:08 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:07.668996+0000 mon.smithi003 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.137:0/2181710149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:09.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:09.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:10.284 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 -- 172.21.15.137:0/2244717605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 msgr2=0x7f03bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:10.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 --2- 172.21.15.137:0/2244717605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f03b0009fb0 tx=0x7f03b0009b30).stop 2023-11-15T06:03:10.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 -- 172.21.15.137:0/2244717605 shutdown_connections 2023-11-15T06:03:10.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 --2- 172.21.15.137:0/2244717605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:10.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 -- 172.21.15.137:0/2244717605 >> 172.21.15.137:0/2244717605 conn(0x7f03bc0f0b70 msgr2=0x7f03bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:10.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 -- 172.21.15.137:0/2244717605 shutdown_connections 2023-11-15T06:03:10.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.282+0000 7f03c1425700 1 -- 172.21.15.137:0/2244717605 wait complete. 2023-11-15T06:03:10.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03c1425700 1 Processor -- start 2023-11-15T06:03:10.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03c1425700 1 -- start start 2023-11-15T06:03:10.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03c1425700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:10.287 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03c1425700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc106c00 con 0x7f03bc0f5410 2023-11-15T06:03:10.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03baffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:10.288 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03baffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47986/0 (socket says 172.21.15.137:47986) 2023-11-15T06:03:10.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.283+0000 7f03baffd700 1 -- 172.21.15.137:0/3128755327 learned_addr learned my addr 172.21.15.137:0/3128755327 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:10.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03baffd700 1 -- 172.21.15.137:0/3128755327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b0005040 con 0x7f03bc0f5410 2023-11-15T06:03:10.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03baffd700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f03b0013510 tx=0x7f03b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:10.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b0009320 con 0x7f03bc0f5410 2023-11-15T06:03:10.289 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03b0013f00 con 0x7f03bc0f5410 2023-11-15T06:03:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03b0014d90 con 0x7f03bc0f5410 2023-11-15T06:03:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc106e00 con 0x7f03bc0f5410 2023-11-15T06:03:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.284+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc107290 con 0x7f03bc0f5410 2023-11-15T06:03:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.285+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc045ad0 con 0x7f03bc0f5410 2023-11-15T06:03:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.286+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03b00095f0 con 0x7f03bc0f5410 2023-11-15T06:03:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.286+0000 7f03a37fe700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 0x7f03a4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.286+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03b00410c0 con 0x7f03bc0f5410 2023-11-15T06:03:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.286+0000 7f03ba7fc700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 0x7f03a4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.286+0000 7f03ba7fc700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 0x7f03a4032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:10.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.290+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03b0014420 con 0x7f03bc0f5410 2023-11-15T06:03:10.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:10 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:09.046861+0000 mgr.smithi003.cldzke (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:10.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.494+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03bc108080 con 0x7f03bc0f5410 2023-11-15T06:03:10.496 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.495+0000 7f03a37fe700 1 -- 172.21.15.137:0/3128755327 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f03b0010230 con 0x7f03bc0f5410 2023-11-15T06:03:10.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:10.497 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:10.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 msgr2=0x7f03a4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:10.498 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 0x7f03a4032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).stop 2023-11-15T06:03:10.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 msgr2=0x7f03bc106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:10.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f03b0013510 tx=0x7f03b000b040).stop 2023-11-15T06:03:10.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 shutdown_connections 2023-11-15T06:03:10.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f03a402ffb0 0x7f03a4032470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:10.499 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.496+0000 7f03c1425700 1 --2- 172.21.15.137:0/3128755327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03bc0f5410 0x7f03bc106710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:10.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.497+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 >> 172.21.15.137:0/3128755327 conn(0x7f03bc0f0b70 msgr2=0x7f03bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:10.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.497+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 shutdown_connections 2023-11-15T06:03:10.500 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:10.497+0000 7f03c1425700 1 -- 172.21.15.137:0/3128755327 wait complete. 2023-11-15T06:03:10.500 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:11.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:11 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:10.496392+0000 mon.smithi003 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.137:0/3128755327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:11.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:11.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:12.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:12 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:11.047216+0000 mgr.smithi003.cldzke (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:13.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 -- 172.21.15.137:0/1480629508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 msgr2=0x7fc3500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:13.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 --2- 172.21.15.137:0/1480629508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc3500f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fc344009fb0 tx=0x7fc344009b30).stop 2023-11-15T06:03:13.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 -- 172.21.15.137:0/1480629508 shutdown_connections 2023-11-15T06:03:13.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 --2- 172.21.15.137:0/1480629508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc3500f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:13.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 -- 172.21.15.137:0/1480629508 >> 172.21.15.137:0/1480629508 conn(0x7fc3500f0b70 msgr2=0x7fc3500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:13.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 -- 172.21.15.137:0/1480629508 shutdown_connections 2023-11-15T06:03:13.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.075+0000 7fc357126700 1 -- 172.21.15.137:0/1480629508 wait complete. 2023-11-15T06:03:13.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc357126700 1 Processor -- start 2023-11-15T06:03:13.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc357126700 1 -- start start 2023-11-15T06:03:13.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc357126700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:13.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc357126700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350108e00 con 0x7fc3500f5410 2023-11-15T06:03:13.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc354ec2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:13.081 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc354ec2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48008/0 (socket says 172.21.15.137:48008) 2023-11-15T06:03:13.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.076+0000 7fc354ec2700 1 -- 172.21.15.137:0/3905348407 learned_addr learned my addr 172.21.15.137:0/3905348407 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:13.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc354ec2700 1 -- 172.21.15.137:0/3905348407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344005040 con 0x7fc3500f5410 2023-11-15T06:03:13.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc354ec2700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fc344013510 tx=0x7fc34400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:13.082 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc344009320 con 0x7fc3500f5410 2023-11-15T06:03:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc350109000 con 0x7fc3500f5410 2023-11-15T06:03:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc344013f00 con 0x7fc3500f5410 2023-11-15T06:03:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc344014d90 con 0x7fc3500f5410 2023-11-15T06:03:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.077+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc350109490 con 0x7fc3500f5410 2023-11-15T06:03:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.078+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc350045ad0 con 0x7fc3500f5410 2023-11-15T06:03:13.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.078+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3440095f0 con 0x7fc3500f5410 2023-11-15T06:03:13.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.078+0000 7fc34d7fa700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 0x7fc340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:13.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.078+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc344040fe0 con 0x7fc3500f5410 2023-11-15T06:03:13.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.078+0000 7fc34ffff700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 0x7fc340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:13.084 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.079+0000 7fc34ffff700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 0x7fc340032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc33c0096a0 tx=0x7fc33c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:13.085 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.083+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc344014420 con 0x7fc3500f5410 2023-11-15T06:03:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.288+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3500fae60 con 0x7fc3500f5410 2023-11-15T06:03:13.290 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.289+0000 7fc34d7fa700 1 -- 172.21.15.137:0/3905348407 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc344010230 con 0x7fc3500f5410 2023-11-15T06:03:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:13.291 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 msgr2=0x7fc340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 0x7fc340032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc33c0096a0 tx=0x7fc33c006b40).stop 2023-11-15T06:03:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 msgr2=0x7fc350108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:13.292 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fc344013510 tx=0x7fc34400b040).stop 2023-11-15T06:03:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 shutdown_connections 2023-11-15T06:03:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fc34002ff60 0x7fc340032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 --2- 172.21.15.137:0/3905348407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3500f5410 0x7fc350108910 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 >> 172.21.15.137:0/3905348407 conn(0x7fc3500f0b70 msgr2=0x7fc35005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:13.293 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 shutdown_connections 2023-11-15T06:03:13.294 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:13.290+0000 7fc357126700 1 -- 172.21.15.137:0/3905348407 wait complete. 2023-11-15T06:03:13.294 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:14.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:13.047612+0000 mgr.smithi003.cldzke (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:14.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:14 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:13.290125+0000 mon.smithi003 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.137:0/3905348407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:14.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:14.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:15.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.810+0000 7ff338751700 1 -- 172.21.15.137:0/3109402014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 msgr2=0x7ff3300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:15.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.810+0000 7ff338751700 1 --2- 172.21.15.137:0/3109402014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3300f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7ff320009fb0 tx=0x7ff320009b30).stop 2023-11-15T06:03:15.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- 172.21.15.137:0/3109402014 shutdown_connections 2023-11-15T06:03:15.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 --2- 172.21.15.137:0/3109402014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3300f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:15.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- 172.21.15.137:0/3109402014 >> 172.21.15.137:0/3109402014 conn(0x7ff3300f0b70 msgr2=0x7ff3300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:15.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- 172.21.15.137:0/3109402014 shutdown_connections 2023-11-15T06:03:15.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- 172.21.15.137:0/3109402014 wait complete. 2023-11-15T06:03:15.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 Processor -- start 2023-11-15T06:03:15.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- start start 2023-11-15T06:03:15.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.811+0000 7ff338751700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff330108dc0 con 0x7ff3300f5410 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff3364ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff3364ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48036/0 (socket says 172.21.15.137:48036) 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff3364ed700 1 -- 172.21.15.137:0/142382460 learned_addr learned my addr 172.21.15.137:0/142382460 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff3364ed700 1 -- 172.21.15.137:0/142382460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320005040 con 0x7ff3300f5410 2023-11-15T06:03:15.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff3364ed700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff320013510 tx=0x7ff32000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:15.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff320009320 con 0x7ff3300f5410 2023-11-15T06:03:15.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff330108fc0 con 0x7ff3300f5410 2023-11-15T06:03:15.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff330109450 con 0x7ff3300f5410 2023-11-15T06:03:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff320013f00 con 0x7ff3300f5410 2023-11-15T06:03:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.812+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff320014d60 con 0x7ff3300f5410 2023-11-15T06:03:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.813+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3200095f0 con 0x7ff3300f5410 2023-11-15T06:03:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.813+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff330045ad0 con 0x7ff3300f5410 2023-11-15T06:03:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.813+0000 7ff326ffd700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 0x7ff31c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:15.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.813+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3200410b0 con 0x7ff3300f5410 2023-11-15T06:03:15.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.814+0000 7ff335cec700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 0x7ff31c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:15.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.814+0000 7ff335cec700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 0x7ff31c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff32c0096a0 tx=0x7ff32c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:15.820 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:15.819+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff320014420 con 0x7ff3300f5410 2023-11-15T06:03:16.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.024+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3300fae60 con 0x7ff3300f5410 2023-11-15T06:03:16.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.025+0000 7ff326ffd700 1 -- 172.21.15.137:0/142382460 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff320012980 con 0x7ff3300f5410 2023-11-15T06:03:16.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:16.027 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:16.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.026+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 msgr2=0x7ff31c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:16.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 0x7ff31c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff32c0096a0 tx=0x7ff32c006b40).stop 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 msgr2=0x7ff3301088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ff320013510 tx=0x7ff32000b040).stop 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 shutdown_connections 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7ff31c02ff60 0x7ff31c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 --2- 172.21.15.137:0/142382460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3300f5410 0x7ff3301088d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:16.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 >> 172.21.15.137:0/142382460 conn(0x7ff3300f0b70 msgr2=0x7ff33005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:16.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 shutdown_connections 2023-11-15T06:03:16.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:16.027+0000 7ff338751700 1 -- 172.21.15.137:0/142382460 wait complete. 2023-11-15T06:03:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:15.047894+0000 mgr.smithi003.cldzke (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:16.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:16 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:16.026270+0000 mon.smithi003 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.137:0/142382460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:17.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:17.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:18.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:18 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:17.048191+0000 mgr.smithi003.cldzke (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:18.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 -- 172.21.15.137:0/934784986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 msgr2=0x7fb5000f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:18.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 --2- 172.21.15.137:0/934784986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5000f6e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb4f0009fb0 tx=0x7fb4f0009b30).stop 2023-11-15T06:03:18.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 -- 172.21.15.137:0/934784986 shutdown_connections 2023-11-15T06:03:18.545 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 --2- 172.21.15.137:0/934784986 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5000f6e20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 -- 172.21.15.137:0/934784986 >> 172.21.15.137:0/934784986 conn(0x7fb5000f0260 msgr2=0x7fb5000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 -- 172.21.15.137:0/934784986 shutdown_connections 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.541+0000 7fb507abc700 1 -- 172.21.15.137:0/934784986 wait complete. 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb507abc700 1 Processor -- start 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb507abc700 1 -- start start 2023-11-15T06:03:18.546 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb507abc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:18.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb507abc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5000f5290 con 0x7fb5000f6a00 2023-11-15T06:03:18.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb505858700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:18.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb505858700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:48064/0 (socket says 172.21.15.137:48064) 2023-11-15T06:03:18.547 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb505858700 1 -- 172.21.15.137:0/2186995166 learned_addr learned my addr 172.21.15.137:0/2186995166 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:18.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.542+0000 7fb505858700 1 -- 172.21.15.137:0/2186995166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0005040 con 0x7fb5000f6a00 2023-11-15T06:03:18.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb505858700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb4f0009830 tx=0x7fb4f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:18.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f0009320 con 0x7fb5000f6a00 2023-11-15T06:03:18.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5000f5490 con 0x7fb5000f6a00 2023-11-15T06:03:18.548 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f00136c0 con 0x7fb5000f6a00 2023-11-15T06:03:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f001ee90 con 0x7fb5000f6a00 2023-11-15T06:03:18.549 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.543+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb500109a80 con 0x7fb5000f6a00 2023-11-15T06:03:18.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4f001e420 con 0x7fb5000f6a00 2023-11-15T06:03:18.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb500059100 con 0x7fb5000f6a00 2023-11-15T06:03:18.550 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb4f67fc700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 0x7fb4ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:18.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4f0041010 con 0x7fb5000f6a00 2023-11-15T06:03:18.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb505057700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 0x7fb4ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:18.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.544+0000 7fb505057700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 0x7fb4ec0323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb4fc0096a0 tx=0x7fb4fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:18.551 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.549+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4f00095f0 con 0x7fb5000f6a00 2023-11-15T06:03:18.763 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.762+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb500045ad0 con 0x7fb5000f6a00 2023-11-15T06:03:18.764 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.762+0000 7fb4f67fc700 1 -- 172.21.15.137:0/2186995166 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb4f001cb10 con 0x7fb5000f6a00 2023-11-15T06:03:18.764 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:18.765 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:18.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 msgr2=0x7fb4ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:18.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 0x7fb4ec0323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb4fc0096a0 tx=0x7fb4fc006b40).stop 2023-11-15T06:03:18.766 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 msgr2=0x7fb5001093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fb4f0009830 tx=0x7fb4f000b040).stop 2023-11-15T06:03:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 shutdown_connections 2023-11-15T06:03:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb4ec02ff10 0x7fb4ec0323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 --2- 172.21.15.137:0/2186995166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5000f6a00 0x7fb5001093d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:18.767 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 >> 172.21.15.137:0/2186995166 conn(0x7fb5000f0260 msgr2=0x7fb5000f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 shutdown_connections 2023-11-15T06:03:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:18.764+0000 7fb507abc700 1 -- 172.21.15.137:0/2186995166 wait complete. 2023-11-15T06:03:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:19.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:19 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:18.763901+0000 mon.smithi003 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.137:0/2186995166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:20.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:20.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:20.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:20 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:19.048577+0000 mgr.smithi003.cldzke (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:21.296 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 -- 172.21.15.137:0/3483277699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 msgr2=0x7f4c580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:21.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 --2- 172.21.15.137:0/3483277699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c580f90f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f4c48009fb0 tx=0x7f4c48009b30).stop 2023-11-15T06:03:21.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 -- 172.21.15.137:0/3483277699 shutdown_connections 2023-11-15T06:03:21.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 --2- 172.21.15.137:0/3483277699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c580f90f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:21.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 -- 172.21.15.137:0/3483277699 >> 172.21.15.137:0/3483277699 conn(0x7f4c580f0b70 msgr2=0x7f4c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:21.297 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 -- 172.21.15.137:0/3483277699 shutdown_connections 2023-11-15T06:03:21.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 -- 172.21.15.137:0/3483277699 wait complete. 2023-11-15T06:03:21.298 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.294+0000 7f4c5e358700 1 Processor -- start 2023-11-15T06:03:21.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c5e358700 1 -- start start 2023-11-15T06:03:21.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c5e358700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:21.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c5e358700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58106bb0 con 0x7f4c58060280 2023-11-15T06:03:21.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:21.300 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58332/0 (socket says 172.21.15.137:58332) 2023-11-15T06:03:21.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c57fff700 1 -- 172.21.15.137:0/2201702642 learned_addr learned my addr 172.21.15.137:0/2201702642 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:21.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.295+0000 7f4c57fff700 1 -- 172.21.15.137:0/2201702642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48005040 con 0x7f4c58060280 2023-11-15T06:03:21.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c57fff700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f4c48013510 tx=0x7f4c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:21.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c48009320 con 0x7f4c58060280 2023-11-15T06:03:21.301 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c58106db0 con 0x7f4c58060280 2023-11-15T06:03:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c48013f00 con 0x7f4c58060280 2023-11-15T06:03:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c48014dc0 con 0x7f4c58060280 2023-11-15T06:03:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.296+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c58107240 con 0x7f4c58060280 2023-11-15T06:03:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c480095f0 con 0x7f4c58060280 2023-11-15T06:03:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58045ad0 con 0x7f4c58060280 2023-11-15T06:03:21.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c54ff9700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 0x7f4c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:21.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c48041000 con 0x7f4c58060280 2023-11-15T06:03:21.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c577fe700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 0x7f4c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:21.303 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.297+0000 7f4c577fe700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 0x7f4c40032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4c4c0096a0 tx=0x7f4c4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:21.304 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.302+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c48014420 con 0x7f4c58060280 2023-11-15T06:03:21.511 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.509+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c581080f0 con 0x7f4c58060280 2023-11-15T06:03:21.512 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.510+0000 7f4c54ff9700 1 -- 172.21.15.137:0/2201702642 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4c48010230 con 0x7f4c58060280 2023-11-15T06:03:21.512 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:21.512 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 msgr2=0x7f4c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 0x7f4c40032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4c4c0096a0 tx=0x7f4c4c006b40).stop 2023-11-15T06:03:21.513 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 msgr2=0x7f4c581066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f4c48013510 tx=0x7f4c4800b040).stop 2023-11-15T06:03:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 shutdown_connections 2023-11-15T06:03:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f4c4002ff60 0x7f4c40032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 --2- 172.21.15.137:0/2201702642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c58060280 0x7f4c581066c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 >> 172.21.15.137:0/2201702642 conn(0x7f4c580f0b70 msgr2=0x7f4c580f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 shutdown_connections 2023-11-15T06:03:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:21.511+0000 7f4c5e358700 1 -- 172.21.15.137:0/2201702642 wait complete. 2023-11-15T06:03:21.515 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:21.048871+0000 mgr.smithi003.cldzke (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:21.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:21 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:21.511298+0000 mon.smithi003 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.137:0/2201702642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:22.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:22.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:24.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 -- 172.21.15.137:0/814597121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 msgr2=0x7fe1140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:24.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 --2- 172.21.15.137:0/814597121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe1140f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe104009fb0 tx=0x7fe104009b30).stop 2023-11-15T06:03:24.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 -- 172.21.15.137:0/814597121 shutdown_connections 2023-11-15T06:03:24.071 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 --2- 172.21.15.137:0/814597121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe1140f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:24.072 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 -- 172.21.15.137:0/814597121 >> 172.21.15.137:0/814597121 conn(0x7fe1140f1c70 msgr2=0x7fe1140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:24.073 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 -- 172.21.15.137:0/814597121 shutdown_connections 2023-11-15T06:03:24.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.067+0000 7fe11b452700 1 -- 172.21.15.137:0/814597121 wait complete. 2023-11-15T06:03:24.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.068+0000 7fe11b452700 1 Processor -- start 2023-11-15T06:03:24.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.068+0000 7fe11b452700 1 -- start start 2023-11-15T06:03:24.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.068+0000 7fe11b452700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:24.074 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.068+0000 7fe11b452700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114069e20 con 0x7fe1140f6510 2023-11-15T06:03:24.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe1191ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:24.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe1191ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58350/0 (socket says 172.21.15.137:58350) 2023-11-15T06:03:24.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe1191ee700 1 -- 172.21.15.137:0/3208076342 learned_addr learned my addr 172.21.15.137:0/3208076342 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:24.075 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe1191ee700 1 -- 172.21.15.137:0/3208076342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104005040 con 0x7fe1140f6510 2023-11-15T06:03:24.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe1191ee700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:24.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104009320 con 0x7fe1140f6510 2023-11-15T06:03:24.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe114069fc0 con 0x7fe1140f6510 2023-11-15T06:03:24.076 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe104013f00 con 0x7fe1140f6510 2023-11-15T06:03:24.077 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe104014d90 con 0x7fe1140f6510 2023-11-15T06:03:24.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.069+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11406a450 con 0x7fe1140f6510 2023-11-15T06:03:24.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.070+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1040095f0 con 0x7fe1140f6510 2023-11-15T06:03:24.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.070+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe114059100 con 0x7fe1140f6510 2023-11-15T06:03:24.078 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.070+0000 7fe109ffb700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 0x7fe100032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:24.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.070+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1040410c0 con 0x7fe1140f6510 2023-11-15T06:03:24.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.071+0000 7fe1189ed700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 0x7fe100032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:24.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.071+0000 7fe1189ed700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 0x7fe100032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe1100096a0 tx=0x7fe110006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:24.079 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.075+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe104014420 con 0x7fe1140f6510 2023-11-15T06:03:24.282 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.281+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1140fbf60 con 0x7fe1140f6510 2023-11-15T06:03:24.283 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.281+0000 7fe109ffb700 1 -- 172.21.15.137:0/3208076342 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe10401d7f0 con 0x7fe1140f6510 2023-11-15T06:03:24.283 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:24.284 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 msgr2=0x7fe100032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 0x7fe100032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe1100096a0 tx=0x7fe110006b40).stop 2023-11-15T06:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 msgr2=0x7fe11406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe104013510 tx=0x7fe10400b040).stop 2023-11-15T06:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 shutdown_connections 2023-11-15T06:03:24.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe10002ff60 0x7fe100032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:24.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 --2- 172.21.15.137:0/3208076342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1140f6510 0x7fe11406b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:24.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 >> 172.21.15.137:0/3208076342 conn(0x7fe1140f1c70 msgr2=0x7fe1140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:24.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 shutdown_connections 2023-11-15T06:03:24.286 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:24.283+0000 7fe11b452700 1 -- 172.21.15.137:0/3208076342 wait complete. 2023-11-15T06:03:24.287 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:24.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:24 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:23.049083+0000 mgr.smithi003.cldzke (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:25.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:25 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:24.282734+0000 mon.smithi003 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.137:0/3208076342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:25.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:25.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:26.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:26 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:25.049391+0000 mgr.smithi003.cldzke (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 -- 172.21.15.137:0/3845432559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 msgr2=0x7fbd080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 --2- 172.21.15.137:0/3845432559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd080f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fbcfc009fb0 tx=0x7fbcfc009b30).stop 2023-11-15T06:03:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 -- 172.21.15.137:0/3845432559 shutdown_connections 2023-11-15T06:03:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 --2- 172.21.15.137:0/3845432559 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd080f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 -- 172.21.15.137:0/3845432559 >> 172.21.15.137:0/3845432559 conn(0x7fbd080f0b70 msgr2=0x7fbd080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 -- 172.21.15.137:0/3845432559 shutdown_connections 2023-11-15T06:03:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.808+0000 7fbd0d479700 1 -- 172.21.15.137:0/3845432559 wait complete. 2023-11-15T06:03:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd0d479700 1 Processor -- start 2023-11-15T06:03:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd0d479700 1 -- start start 2023-11-15T06:03:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd0d479700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd0d479700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd08108e00 con 0x7fbd080f5410 2023-11-15T06:03:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd06ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd06ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:58360/0 (socket says 172.21.15.137:58360) 2023-11-15T06:03:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.809+0000 7fbd06ffd700 1 -- 172.21.15.137:0/930351911 learned_addr learned my addr 172.21.15.137:0/930351911 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbd06ffd700 1 -- 172.21.15.137:0/930351911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc005040 con 0x7fbd080f5410 2023-11-15T06:03:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbd06ffd700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fbcfc013510 tx=0x7fbcfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcfc009320 con 0x7fbd080f5410 2023-11-15T06:03:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd08109000 con 0x7fbd080f5410 2023-11-15T06:03:26.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcfc013f00 con 0x7fbd080f5410 2023-11-15T06:03:26.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcfc014d90 con 0x7fbd080f5410 2023-11-15T06:03:26.816 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.810+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd08109490 con 0x7fbd080f5410 2023-11-15T06:03:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.811+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcfc014420 con 0x7fbd080f5410 2023-11-15T06:03:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.811+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd08045ad0 con 0x7fbd080f5410 2023-11-15T06:03:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.811+0000 7fbcef7fe700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 0x7fbcf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.811+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcfc0410c0 con 0x7fbd080f5410 2023-11-15T06:03:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.811+0000 7fbd067fc700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 0x7fbcf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.812+0000 7fbd067fc700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 0x7fbcf0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbcf80096a0 tx=0x7fbcf8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:26.816+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcfc0095f0 con 0x7fbd080f5410 2023-11-15T06:03:27.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.024+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd080fae60 con 0x7fbd080f5410 2023-11-15T06:03:27.026 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.024+0000 7fbcef7fe700 1 -- 172.21.15.137:0/930351911 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbcfc01db70 con 0x7fbd080f5410 2023-11-15T06:03:27.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:27.027 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:27.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 msgr2=0x7fbcf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:27.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 0x7fbcf0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fbcf80096a0 tx=0x7fbcf8006b40).stop 2023-11-15T06:03:27.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 msgr2=0x7fbd08108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:27.028 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fbcfc013510 tx=0x7fbcfc00b040).stop 2023-11-15T06:03:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 shutdown_connections 2023-11-15T06:03:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fbcf002ff60 0x7fbcf0032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 --2- 172.21.15.137:0/930351911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbd080f5410 0x7fbd08108910 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 >> 172.21.15.137:0/930351911 conn(0x7fbd080f0b70 msgr2=0x7fbd0805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 shutdown_connections 2023-11-15T06:03:27.030 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:27.026+0000 7fbd0d479700 1 -- 172.21.15.137:0/930351911 wait complete. 2023-11-15T06:03:27.030 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:27.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:27 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:27.026004+0000 mon.smithi003 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.137:0/930351911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:28.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:28.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:28.430 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:28 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:27.049670+0000 mgr.smithi003.cldzke (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:29.580 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 -- 172.21.15.137:0/285297585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 msgr2=0x7f48780fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:29.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 --2- 172.21.15.137:0/285297585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48780fb570 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4868009fb0 tx=0x7f4868009b30).stop 2023-11-15T06:03:29.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 -- 172.21.15.137:0/285297585 shutdown_connections 2023-11-15T06:03:29.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 --2- 172.21.15.137:0/285297585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48780fb570 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:29.581 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 -- 172.21.15.137:0/285297585 >> 172.21.15.137:0/285297585 conn(0x7f48780f0b70 msgr2=0x7f48780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:29.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 -- 172.21.15.137:0/285297585 shutdown_connections 2023-11-15T06:03:29.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 -- 172.21.15.137:0/285297585 wait complete. 2023-11-15T06:03:29.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.578+0000 7f487e7b0700 1 Processor -- start 2023-11-15T06:03:29.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f487e7b0700 1 -- start start 2023-11-15T06:03:29.582 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f487e7b0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f487e7b0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4878108dc0 con 0x7f4878060200 2023-11-15T06:03:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f4877fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f4877fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47888/0 (socket says 172.21.15.137:47888) 2023-11-15T06:03:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f4877fff700 1 -- 172.21.15.137:0/3940507905 learned_addr learned my addr 172.21.15.137:0/3940507905 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f4877fff700 1 -- 172.21.15.137:0/3940507905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868005040 con 0x7f4878060200 2023-11-15T06:03:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.579+0000 7f4877fff700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4868013510 tx=0x7f486800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4868009320 con 0x7f4878060200 2023-11-15T06:03:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4878108fc0 con 0x7f4878060200 2023-11-15T06:03:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4868013f00 con 0x7f4878060200 2023-11-15T06:03:29.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4868014dc0 con 0x7f4878060200 2023-11-15T06:03:29.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4878109450 con 0x7f4878060200 2023-11-15T06:03:29.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.580+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4868014420 con 0x7f4878060200 2023-11-15T06:03:29.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.581+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4878045ad0 con 0x7f4878060200 2023-11-15T06:03:29.586 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.581+0000 7f4874ff9700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 0x7f4860032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:29.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.581+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4868041040 con 0x7f4878060200 2023-11-15T06:03:29.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.581+0000 7f48777fe700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 0x7f4860032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:29.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.581+0000 7f48777fe700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 0x7f4860032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f486c0096a0 tx=0x7f486c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:29.587 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.585+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48680095f0 con 0x7f4878060200 2023-11-15T06:03:29.795 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.794+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f487810a2a0 con 0x7f4878060200 2023-11-15T06:03:29.796 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.794+0000 7f4874ff9700 1 -- 172.21.15.137:0/3940507905 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f486801cb10 con 0x7f4878060200 2023-11-15T06:03:29.796 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:29.796 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:29.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 msgr2=0x7f4860032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:29.797 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 0x7f4860032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f486c0096a0 tx=0x7f486c006b40).stop 2023-11-15T06:03:29.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 msgr2=0x7f48781088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:29.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f4868013510 tx=0x7f486800b040).stop 2023-11-15T06:03:29.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 shutdown_connections 2023-11-15T06:03:29.798 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f486002ff60 0x7f4860032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:29.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 --2- 172.21.15.137:0/3940507905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4878060200 0x7f48781088d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:29.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 >> 172.21.15.137:0/3940507905 conn(0x7f48780f0b70 msgr2=0x7f48780f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:29.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 shutdown_connections 2023-11-15T06:03:29.799 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:29.796+0000 7f487e7b0700 1 -- 172.21.15.137:0/3940507905 wait complete. 2023-11-15T06:03:29.800 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:29.050028+0000 mgr.smithi003.cldzke (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:30.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:30 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:29.795720+0000 mon.smithi003 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.137:0/3940507905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:31.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:31.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:31.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:31 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:31.050395+0000 mgr.smithi003.cldzke (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:32.361 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/4289443791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e00f5410 msgr2=0x7fb5e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:32.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/4289443791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e00f5410 0x7fb5e00f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fb5d0009fb0 tx=0x7fb5d0009b30).stop 2023-11-15T06:03:32.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/4289443791 shutdown_connections 2023-11-15T06:03:32.362 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/4289443791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e00f5410 0x7fb5e00f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:32.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/4289443791 >> 172.21.15.137:0/4289443791 conn(0x7fb5e00f0b70 msgr2=0x7fb5e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:32.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/4289443791 shutdown_connections 2023-11-15T06:03:32.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.358+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/4289443791 wait complete. 2023-11-15T06:03:32.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e7f3c700 1 Processor -- start 2023-11-15T06:03:32.363 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e7f3c700 1 -- start start 2023-11-15T06:03:32.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e7f3c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:32.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e7f3c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0106050 con 0x7fb5e0105740 2023-11-15T06:03:32.364 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e5cd8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:32.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e5cd8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47912/0 (socket says 172.21.15.137:47912) 2023-11-15T06:03:32.365 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e5cd8700 1 -- 172.21.15.137:0/1436185320 learned_addr learned my addr 172.21.15.137:0/1436185320 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:32.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e5cd8700 1 -- 172.21.15.137:0/1436185320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0005040 con 0x7fb5e0105740 2023-11-15T06:03:32.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.359+0000 7fb5e5cd8700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb5d0009f80 tx=0x7fb5d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:32.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d00095f0 con 0x7fb5e0105740 2023-11-15T06:03:32.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5d0007e70 con 0x7fb5e0105740 2023-11-15T06:03:32.366 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e0106250 con 0x7fb5e0105740 2023-11-15T06:03:32.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d001fd90 con 0x7fb5e0105740 2023-11-15T06:03:32.367 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0106da0 con 0x7fb5e0105740 2023-11-15T06:03:32.368 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.360+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5d00079c0 con 0x7fb5e0105740 2023-11-15T06:03:32.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.361+0000 7fb5d67fc700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:32.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.361+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5d0043330 con 0x7fb5e0105740 2023-11-15T06:03:32.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.361+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0059100 con 0x7fb5e0105740 2023-11-15T06:03:32.370 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.361+0000 7fb5e54d7700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:32.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.361+0000 7fb5e54d7700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 0x7fb5cc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb5dc0096a0 tx=0x7fb5dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:32.371 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.366+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5d0014420 con 0x7fb5e0105740 2023-11-15T06:03:32.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.571+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5e01074a0 con 0x7fb5e0105740 2023-11-15T06:03:32.573 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.572+0000 7fb5d67fc700 1 -- 172.21.15.137:0/1436185320 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb5d0027390 con 0x7fb5e0105740 2023-11-15T06:03:32.574 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:32.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:32.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 msgr2=0x7fb5cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:32.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 0x7fb5cc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb5dc0096a0 tx=0x7fb5dc006b40).stop 2023-11-15T06:03:32.575 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 msgr2=0x7fb5e0105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:32.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb5d0009f80 tx=0x7fb5d0009b30).stop 2023-11-15T06:03:32.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 shutdown_connections 2023-11-15T06:03:32.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:32.576 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 --2- 172.21.15.137:0/1436185320 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5e0105740 0x7fb5e0105b60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:32.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 >> 172.21.15.137:0/1436185320 conn(0x7fb5e00f0b70 msgr2=0x7fb5e00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:32.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 shutdown_connections 2023-11-15T06:03:32.577 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:32.573+0000 7fb5e7f3c700 1 -- 172.21.15.137:0/1436185320 wait complete. 2023-11-15T06:03:32.578 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:32.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:32 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:32.573236+0000 mon.smithi003 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.137:0/1436185320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:33.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:33 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:33.050764+0000 mgr.smithi003.cldzke (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:33.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:33.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.140+0000 7f6982857700 1 -- 172.21.15.137:0/196087750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 msgr2=0x7f697c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.140+0000 7f6982857700 1 --2- 172.21.15.137:0/196087750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c0f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f696c009fb0 tx=0x7f696c009b30).stop 2023-11-15T06:03:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 -- 172.21.15.137:0/196087750 shutdown_connections 2023-11-15T06:03:35.143 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 --2- 172.21.15.137:0/196087750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c0f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:35.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 -- 172.21.15.137:0/196087750 >> 172.21.15.137:0/196087750 conn(0x7f697c0f1c70 msgr2=0x7f697c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:35.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 -- 172.21.15.137:0/196087750 shutdown_connections 2023-11-15T06:03:35.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 -- 172.21.15.137:0/196087750 wait complete. 2023-11-15T06:03:35.144 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 Processor -- start 2023-11-15T06:03:35.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.141+0000 7f6982857700 1 -- start start 2023-11-15T06:03:35.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f6982857700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:35.146 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f6982857700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c106c20 con 0x7f697c0f6510 2023-11-15T06:03:35.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f697bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:35.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f697bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47932/0 (socket says 172.21.15.137:47932) 2023-11-15T06:03:35.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f697bfff700 1 -- 172.21.15.137:0/3286070278 learned_addr learned my addr 172.21.15.137:0/3286070278 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:35.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f697bfff700 1 -- 172.21.15.137:0/3286070278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c005040 con 0x7f697c0f6510 2023-11-15T06:03:35.147 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.142+0000 7f697bfff700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f696c013510 tx=0x7f696c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:35.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696c009320 con 0x7f697c0f6510 2023-11-15T06:03:35.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c106e20 con 0x7f697c0f6510 2023-11-15T06:03:35.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f696c013f00 con 0x7f697c0f6510 2023-11-15T06:03:35.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f696c014d90 con 0x7f697c0f6510 2023-11-15T06:03:35.148 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c1072b0 con 0x7f697c0f6510 2023-11-15T06:03:35.149 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f696c0095f0 con 0x7f697c0f6510 2023-11-15T06:03:35.149 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.143+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c045ad0 con 0x7f697c0f6510 2023-11-15T06:03:35.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.144+0000 7f6978ff9700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:35.151 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.144+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f696c041000 con 0x7f697c0f6510 2023-11-15T06:03:35.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.144+0000 7f697b7fe700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:35.152 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.144+0000 7f697b7fe700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 0x7f6964032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f69700096a0 tx=0x7f6970006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:35.153 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.148+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f696c014420 con 0x7f697c0f6510 2023-11-15T06:03:35.355 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.353+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f697c0fbf60 con 0x7f697c0f6510 2023-11-15T06:03:35.356 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.354+0000 7f6978ff9700 1 -- 172.21.15.137:0/3286070278 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f696c01d7f0 con 0x7f697c0f6510 2023-11-15T06:03:35.356 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:35.356 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:35.357 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 msgr2=0x7f6964032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:35.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 0x7f6964032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f69700096a0 tx=0x7f6970006b40).stop 2023-11-15T06:03:35.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 msgr2=0x7f697c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:35.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f696c013510 tx=0x7f696c00b040).stop 2023-11-15T06:03:35.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 shutdown_connections 2023-11-15T06:03:35.358 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7f696402ff60 0x7f6964032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:35.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 --2- 172.21.15.137:0/3286070278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f697c0f6510 0x7f697c106730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:35.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 >> 172.21.15.137:0/3286070278 conn(0x7f697c0f1c70 msgr2=0x7f697c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:35.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 shutdown_connections 2023-11-15T06:03:35.359 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:35.356+0000 7f6982857700 1 -- 172.21.15.137:0/3286070278 wait complete. 2023-11-15T06:03:35.361 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:35.051106+0000 mgr.smithi003.cldzke (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:35.119095+0000 mon.smithi003 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:03:36.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:35.121234+0000 mon.smithi003 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.3:0/838558658' entity='mgr.smithi003.cldzke' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.cldzke/trash_purge_schedule"}]: dispatch 2023-11-15T06:03:36.432 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:36 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: audit 2023-11-15T06:03:35.355564+0000 mon.smithi003 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.137:0/3286070278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:03:36.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:03:36.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:03:37.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.992+0000 7fe37223f700 1 -- 172.21.15.137:0/2540444134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 msgr2=0x7fe36c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:37.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.992+0000 7fe37223f700 1 --2- 172.21.15.137:0/2540444134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c0f8f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fe368009fb0 tx=0x7fe368009b30).stop 2023-11-15T06:03:37.996 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- 172.21.15.137:0/2540444134 shutdown_connections 2023-11-15T06:03:37.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 --2- 172.21.15.137:0/2540444134 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c0f8f80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:37.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- 172.21.15.137:0/2540444134 >> 172.21.15.137:0/2540444134 conn(0x7fe36c0f09a0 msgr2=0x7fe36c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:37.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- 172.21.15.137:0/2540444134 shutdown_connections 2023-11-15T06:03:37.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- 172.21.15.137:0/2540444134 wait complete. 2023-11-15T06:03:37.997 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 Processor -- start 2023-11-15T06:03:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- start start 2023-11-15T06:03:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.993+0000 7fe37223f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c1069f0 con 0x7fe36c060180 2023-11-15T06:03:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37123d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37123d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.137:47954/0 (socket says 172.21.15.137:47954) 2023-11-15T06:03:37.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37123d700 1 -- 172.21.15.137:0/1741776003 learned_addr learned my addr 172.21.15.137:0/1741776003 (peer_addr_for_me v2:172.21.15.137:0/0) 2023-11-15T06:03:37.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37123d700 1 -- 172.21.15.137:0/1741776003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368005040 con 0x7fe36c060180 2023-11-15T06:03:37.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37123d700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fe368013510 tx=0x7fe36800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:37.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe368009320 con 0x7fe36c060180 2023-11-15T06:03:37.999 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c106bf0 con 0x7fe36c060180 2023-11-15T06:03:38.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe368013f00 con 0x7fe36c060180 2023-11-15T06:03:38.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe368014d90 con 0x7fe36c060180 2023-11-15T06:03:38.000 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.994+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c107080 con 0x7fe36c060180 2023-11-15T06:03:38.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.995+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3680095f0 con 0x7fe36c060180 2023-11-15T06:03:38.001 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.995+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c045ad0 con 0x7fe36c060180 2023-11-15T06:03:38.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.995+0000 7fe361ffb700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 0x7fe358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:03:38.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.995+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe368040fc0 con 0x7fe36c060180 2023-11-15T06:03:38.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.996+0000 7fe370a3c700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 0x7fe358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:03:38.002 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:37.996+0000 7fe370a3c700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 0x7fe358032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:03:38.003 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.001+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe368014420 con 0x7fe36c060180 2023-11-15T06:03:38.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.206+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe36c107f30 con 0x7fe36c060180 2023-11-15T06:03:38.208 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.206+0000 7fe361ffb700 1 -- 172.21.15.137:0/1741776003 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe368010230 con 0x7fe36c060180 2023-11-15T06:03:38.208 INFO:teuthology.orchestra.run.smithi137.stdout: 2023-11-15T06:03:38.209 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"412cc4c0-837b-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T05:53:35.111800Z","created":"2023-11-15T05:53:35.111800Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:03:38.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 msgr2=0x7fe358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:38.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 0x7fe358032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe35c0096a0 tx=0x7fe35c006b40).stop 2023-11-15T06:03:38.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 msgr2=0x7fe36c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:03:38.210 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fe368013510 tx=0x7fe36800b040).stop 2023-11-15T06:03:38.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 shutdown_connections 2023-11-15T06:03:38.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:6800/4263843884,v1:172.21.15.3:6801/4263843884] conn(0x7fe35802ff60 0x7fe358032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:38.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 --2- 172.21.15.137:0/1741776003 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe36c060180 0x7fe36c106500 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:03:38.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 >> 172.21.15.137:0/1741776003 conn(0x7fe36c0f09a0 msgr2=0x7fe36c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:03:38.211 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 shutdown_connections 2023-11-15T06:03:38.212 INFO:teuthology.orchestra.run.smithi137.stderr:2023-11-15T06:03:38.208+0000 7fe37223f700 1 -- 172.21.15.137:0/1741776003 wait complete. 2023-11-15T06:03:38.212 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2023-11-15T06:03:38.431 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: cluster 2023-11-15T06:03:37.051534+0000 mgr.smithi003.cldzke (mgr.14164) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:03:38.596 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T06:03:38.597 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T06:03:38.597 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:03:38.624 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:03:38.643 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T06:03:38.643 INFO:tasks.cephadm.mon.smithi003:Stopping mon.smithi003... 2023-11-15T06:03:38.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-15T06:03:39.012 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:38 smithi003 systemd[1]: Stopping Ceph mon.smithi003 for 412cc4c0-837b-11ee-8dbe-212e2dc638e7... 2023-11-15T06:03:39.012 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T06:03:38.811+0000 7f910217f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi003 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T06:03:39.012 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 15 06:03:38 smithi003 ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7-mon.smithi003[112385]: debug 2023-11-15T06:03:38.811+0000 7f910217f700 -1 mon.smithi003@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T06:03:39.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi003.service' 2023-11-15T06:03:39.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T06:03:39.166 INFO:tasks.cephadm.mon.smithi003:Stopped mon.smithi003 2023-11-15T06:03:39.166 INFO:tasks.cephadm.mon.smithi137:Stopping mon.smithi137... 2023-11-15T06:03:39.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137 2023-11-15T06:03:39.201 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137.service: Unit ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137.service not loaded. 2023-11-15T06:03:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T06:03:39.203 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi137" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T06:03:39.204 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T06:03:39.207 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/ubuntu@smithi003.front.sepia.ceph.com/crash 2023-11-15T06:03:39.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:03:39.265 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/ubuntu@smithi137.front.sepia.ceph.com/crash 2023-11-15T06:03:39.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:03:39.308 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T06:03:39.308 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2023-11-15T06:03:39.309 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T06:03:39.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T06:03:39.350 INFO:teuthology.orchestra.run.smithi003.stderr:grep: /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T06:03:39.352 INFO:tasks.cephadm:Compressing logs... 2023-11-15T06:03:39.352 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:03:39.394 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:03:39.421 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:03:39.422 INFO:teuthology.orchestra.run.smithi137.stderr:: No such file or directory 2023-11-15T06:03:39.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:03:39.425 INFO:teuthology.orchestra.run.smithi137.stderr: 2023-11-15T06:03:39.425 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m0.013s 2023-11-15T06:03:39.425 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m0.013s 2023-11-15T06:03:39.425 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m0.008s 2023-11-15T06:03:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:03:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2023-11-15T06:03:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T06:03:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.cldzke.log 2023-11-15T06:03:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log 2023-11-15T06:03:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.cldzke.log: /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log: 88.3% -- replaced with /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T06:03:39.433 INFO:teuthology.orchestra.run.smithi003.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:03:39.511 INFO:teuthology.orchestra.run.smithi003.stderr: 92.7% -- replaced with /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.cldzke.log.gz 2023-11-15T06:03:39.582 INFO:teuthology.orchestra.run.smithi003.stderr: 90.6% -- replaced with /var/log/ceph/412cc4c0-837b-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log.gz 2023-11-15T06:03:39.583 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-15T06:03:39.584 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.170s 2023-11-15T06:03:39.584 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m0.237s 2023-11-15T06:03:39.584 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.033s 2023-11-15T06:03:39.585 INFO:tasks.cephadm:Archiving logs... 2023-11-15T06:03:39.585 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/ubuntu@smithi003.front.sepia.ceph.com/log 2023-11-15T06:03:39.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:03:39.704 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/ubuntu@smithi137.front.sepia.ceph.com/log 2023-11-15T06:03:39.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:03:39.734 INFO:tasks.cephadm:Removing cluster... 2023-11-15T06:03:39.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:04:33.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 412cc4c0-837b-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:04:33.350 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T06:04:33.350 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:04:33.375 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:04:33.394 INFO:tasks.cephadm:Teardown complete 2023-11-15T06:04:33.395 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T06:04:33.477 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c060dcc2494542fcba63d7d6d2067d7c Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi137 with status 5: 'sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137' 2023-11-15T06:04:33.482 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T06:04:33.491 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T06:04:33.509 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T06:04:33.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:04:33.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:04:33.615 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T06:04:33.793 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T06:04:33.794 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:04:33.797 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:04:33.820 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-11-15T06:04:33.821 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2023-11-15T06:04:33.821 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:04:33.822 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2023-11-15T06:04:33.822 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +1254us[+1254us] +/- 90ms 2023-11-15T06:04:33.822 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -800us[ -789us] +/- 68ms 2023-11-15T06:04:33.823 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -108us[ -97us] +/- 75ms 2023-11-15T06:04:33.823 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:04:33.824 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:04:33.824 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-11-15T06:04:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +1259us[+1259us] +/- 90ms 2023-11-15T06:04:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -775us[ -775us] +/- 68ms 2023-11-15T06:04:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -100us[ -100us] +/- 75ms 2023-11-15T06:04:33.825 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:04:33.826 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T06:04:33.840 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T06:04:33.841 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T06:04:33.858 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:04:33.972 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:04:34.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:04:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:04:34.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:04:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:04:34.141 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T06:04:34.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T06:04:34.175 INFO:teuthology.task.internal:Duration was 1431.718593 seconds 2023-11-15T06:04:34.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T06:04:34.188 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T06:04:34.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:04:34.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:04:34.246 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:04:34.253 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:04:34.694 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T06:04:34.694 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-11-15T06:04:34.695 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:04:34.724 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2023-11-15T06:04:34.724 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:04:34.748 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T06:04:34.749 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:04:34.768 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:04:34.834 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T06:04:34.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:04:34.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:04:34.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T06:04:34.953 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T06:04:34.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:04:34.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:04:35.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T06:04:35.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:04:35.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:04:35.060 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-11-15T06:04:35.081 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2023-11-15T06:04:35.141 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:04:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:04:35.205 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:04:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:04:35.223 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T06:04:35.234 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T06:04:35.235 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/smithi003 2023-11-15T06:04:35.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:04:35.294 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530/remote/smithi137 2023-11-15T06:04:35.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:04:35.336 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T06:04:35.336 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:04:35.339 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:04:35.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T06:04:35.405 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T06:04:35.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T06:04:35.416 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T06:04:35.417 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:04:35.420 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:04:35.440 INFO:teuthology.orchestra.run.smithi003.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:04 /home/ubuntu/cephtest 2023-11-15T06:04:35.447 INFO:teuthology.orchestra.run.smithi137.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:04 /home/ubuntu/cephtest 2023-11-15T06:04:35.448 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T06:04:35.462 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T06:04:35.486 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T06:04:35.514 DEBUG:teuthology.nuke:shortname: smithi003 2023-11-15T06:04:35.515 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:04:35.540 DEBUG:teuthology.nuke:shortname: smithi137 2023-11-15T06:04:35.541 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:04:35.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 05:31:46.767699', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBOXwiuJfu0mmNPAy+Zd1ksMS5wpRkhqhKpdWlFA/ku+vv509YqaBYDr8EuJECVM8hJLam7DXA2h6UBsHaeUO1E='} 2023-11-15T06:04:35.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 05:31:46.766042', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICrjbq4dUH/QpLH8b3n08i+dpWeh1eYa74dN94LX7sFrf7N/CUHLvmM+nk5Jm/D3t7bTfjpbR6BDNHLpP6AsD0='} 2023-11-15T06:04:35.601 INFO:teuthology.orchestra.console.smithi003:Power off 2023-11-15T06:04:35.603 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:04:35.626 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-11-15T06:04:35.626 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:35.647 INFO:teuthology.orchestra.console.smithi137:Power off 2023-11-15T06:04:35.647 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:04:35.661 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-11-15T06:04:35.674 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2023-11-15T06:04:35.674 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:35.699 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-11-15T06:04:39.663 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:39.701 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:39.775 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-11-15T06:04:39.815 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2023-11-15T06:04:43.778 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:43.816 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:43.892 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-11-15T06:04:43.929 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2023-11-15T06:04:44.030 INFO:teuthology.orchestra.console.smithi137:Power off completed 2023-11-15T06:04:47.894 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:04:48.006 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-11-15T06:04:48.108 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-11-15T06:04:48.213 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1431.7185933589935 failure_reason: 'Command failed on smithi137 with status 5: ''sudo systemctl stop ceph-412cc4c0-837b-11ee-8dbe-212e2dc638e7@mon.smithi137''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c060dcc2494542fcba63d7d6d2067d7c status: fail success: false 2023-11-15T06:04:48.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T06:04:48.300 INFO:teuthology.run:FAIL